git-vqav2-medium
microsoft/git-base-vqav2 fine-tuned on VQAv2 (50,000 training questions).
Model
| Architecture | GitForCausalLM (GIT-base) |
| Vision encoder | CLIP ViT-B/16, 12 layers, 480×480 input, 901 visual tokens |
| Text decoder | 6-layer causal Transformer, hidden 768, 12 heads, max 1024 positions |
| Tokenizer | BERT WordPiece (uncased), vocab 30,522 |
| Parameters | 177,159,738 (trainable during fine-tuning: 177,159,738, strategy full) |
| Weights | fp32, safetensors |
Input / output format
- Image:
CLIPImageProcessor— resize shortest edge to 480 (bicubic), center crop 480×480, rescale 1/255, normalize with CLIP mean/std. - Text prompt:
[CLS] question(no[SEP]between question and answer). - Output: the answer tokens generated after the prompt, terminated by
[SEP](eos id 102). - Training sequence:
[CLS] question answer [SEP]; loss only onanswer [SEP](all other positions-100).
Training data
| Dataset | HuggingFaceM4/VQAv2, parquet conversion refs/convert/parquet @ ac4a1a047466a7019b5357ce94b6b5794fc9f867 |
| Split | train (COCO train2014 images) |
| Shards / questions | [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] / 50,000 |
| Rows skipped (missing question/answer) | 0 |
| Target answer | majority vote over the 10 human answers (lower-cased, whitespace/edge punctuation stripped; ties → multiple_choice_answer) |
| Model selection | 1,000 questions from validation shards [2, 7], best vqa_accuracy |
Training procedure
| Hyperparameter | Value |
|---|---|
| Epochs | 2 |
| Optimizer | AdamW (β=(0.9, 0.999), ε=1e-8), no decay on biases/LayerNorm |
| Learning rate | 2e-06, linear decay, warmup 156 steps |
| Batch size | 8 × 4 accumulation = 32 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| Precision | bf16 autocast, fp32 master weights |
| Vision attention | sdpa |
| Optimizer steps | 3,126 |
| Best checkpoint | step 500 (vqa_accuracy = 84.39) |
| Seed | 42 |
| Hardware | NVIDIA GeForce RTX 5060 Ti |
| Software | torch 2.11.0+cu128, transformers 5.17.0, datasets 4.3.0 |
Evaluation
VQAv2 consensus accuracy (official vqaEval.py normalization; per question mean over 10 leave-one-out subsets of min(1, matches/3)) and exact match against the majority answer (not an official metric). Deterministic decoding: max_new_tokens=10, num_beams=1, do_sample=False.
Evaluation set: 500 questions sampled with a fixed seed from validation shard(s) [0] (subset fingerprint 1daf0de4dae278b4), disjoint from the model-selection questions. Both models were evaluated on exactly the same questions.
| Metric | microsoft/git-base-vqav2 | This model | Δ |
|---|---|---|---|
| VQA accuracy | 83.32 | 82.54 | -0.78 |
| Exact match | 73.40 | 72.00 | -1.40 |
| VQA accuracy (number) | 79.47 | 77.07 | -2.40 |
| VQA accuracy (other) | 75.74 | 74.84 | -0.90 |
| VQA accuracy (yes/no) | 95.14 | 95.19 | +0.05 |
Usage
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM
repo = "thealper2/git-vqav2-medium"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()
image = Image.open("example.jpg").convert("RGB")
question = "what color is the car?"
pixel_values = processor(images=image, return_tensors="pt").pixel_values
input_ids = [processor.tokenizer.cls_token_id] + processor.tokenizer(question, add_special_tokens=False).input_ids
input_ids = torch.tensor([input_ids])
out = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_new_tokens=10)
print(processor.batch_decode(out[:, input_ids.shape[1]:], skip_special_tokens=True)[0])
Batched generation: GIT does not derive text position ids from the attention mask, so padded prompts change the output. Batch only prompts of equal token length.
Limitations
- Scores are computed on a subset of the official validation split, not on the test-dev evaluation server, and are not comparable to published test-dev numbers.
- The base checkpoint scores well above published GIT-base test-dev results on these validation questions, which indicates that VQAv2 validation data was part of its original fine-tuning. Absolute accuracies on validation are therefore optimistic for both models.
- Answers are lower-case (uncased tokenizer). WordPiece splits punctuation, so decoded answers such as
11 : 10need detokenization (11:10). - English only; COCO-domain images.