Training
| base model | openai/clip-vit-base-patch32 |
| objective | symmetric InfoNCE over (image, caption) pairs, using CLIP's learned temperature — the pre-training objective continued at a low learning rate |
| data source | sayakpaul/coco-30-val-2014 (streaming split train) |
| training pairs | 4000 |
| epochs | 2 |
| optimiser steps | 126 |
| batch size | 64 |
| optimiser | AdamW, lr 1e-05, weight decay 0.1 |
| schedule | 10 % linear warmup, then linear decay to 0 |
| gradient clipping | global norm 1.0 |
| seed | 42 |
| device | cuda |
| wall-clock | 345 s |
| loss | 0.4031 → 0.0519 |
Evaluation-set leakage control
The retrieval evaluation uses specific stream indices of sayakpaul/coco-30-val-2014. The training script
streams past those and takes only samples whose index is not in the evaluation set,
and asserts the disjointness at run time rather than assuming it.
Train/eval index overlap = 0 (verified, 500 evaluation indices held out against 4000 training indices).
Both index lists are shipped in finetune_config.json, so the disjointness is checkable
without rerunning anything.
Effect of fine-tuning
Mean cosine similarity on a held-out sample of evaluation pairs, ground-truth versus foil caption:
| zero-shot | fine-tuned | margin change | |
|---|---|---|---|
| MSCOCO | GT 0.3245 / foil 0.1490 | GT 0.3165 / foil 0.0976 | +0.0434 |
Fine-tuning widens the ground-truth-versus-foil margin mainly by pushing foil similarity down.
Usage
from transformers import CLIPModel, CLIPProcessor
model = CLIPModel.from_pretrained("aaronwzl/clip-vit-base-patch32-coco-retrieval")
proc = CLIPProcessor.from_pretrained("aaronwzl/clip-vit-base-patch32-coco-retrieval")
Reproduction
python rebuttal/exp/retrieval/finetune_clip.py \
--dataset coco --n-train 4000 --epochs 2 \
--batch-size 64 --lr 1e-05 --seed 42