Open-ended VQA, trained on train2014 + val2014
The protocol behind VQA v1's published test-dev numbers: train on both annotated splits, evaluate on test-dev. Built on the attention layer of Factor Graph Attention (CVPR 2019).
Code: github.com/idansc/fga.
No score, by construction
VQA v1's test answers were never released — visualqa.org distributes annotations for train2014 and val2014 only — so this model has no number attached to it. One exists only after predictions are submitted to the evaluation server.
For a model with a verifiable score, see Idan/fga-vqa: the same architecture trained on train2014 alone, scoring 61.97 on val2014 under the official metric.
Training
351,596 questions (230,084 train + 121,512 val), 36 bottom-up region features, 20
epochs, batch 512, soft_ce against the VQA score each answer earns from the ten
annotators — worth about a point over a single label, and about 0.8 over the
sigmoid-and-binary-cross-entropy form the 2017 challenge writeup recommends.
Usage
from fga.tasks.vqa import OpenEndedVQAModel
model = OpenEndedVQAModel.from_pretrained("Idan/fga-vqa-trainval")
out = model(question_input_ids=q, image_features=v)
To produce a submission file for the evaluation server:
python scripts/predict_vqa_test.py --model Idan/fga-vqa-trainval --vqa_dir vqa \
--questions vqa/raw/OpenEnded_mscoco_test2015_questions.json \
--features vqa/test_features.h5 --output test2015_results.json
That script tokenizes with the training vocabulary and applies the same per-region feature normalization used during training; getting either wrong quietly corrupts a submission.
Citation
@inproceedings{schwartz2019factor,
title={Factor graph attention},
author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={2039--2048},
year={2019}
}
@inproceedings{schwartz2017high,
title={High-Order Attention Models for Visual Question Answering},
author={Schwartz, Idan and Schwing, Alexander G and Hazan, Tamir},
booktitle={Advances in Neural Information Processing Systems},
year={2017}
}