CLIP ViT-L/14 fine-tuned on 100 image-classification tasks
One fine-tuned vision encoder per task, for model-merging research: <task>/vision_model/ is a
transformers.CLIPVisionModel (save_pretrained), and <task>/result.json holds the training record (split sizes,
zero-shot and fine-tuned val/test accuracy, recipe).
Recipe
Starting from openai/clip-vit-large-patch14, only the vision tower is trained (text tower, visual_projection and
logit_scale frozen). The classifier is the zero-shot head: for each class, the text embeddings of all templates are
L2-normalised, averaged and normalised again; logits = logit_scale.exp() * normalize(image_embeds) @ head.T.
| setting | value |
|---|---|
| optimizer | Adam, lr 1e-5, weight decay 0 |
| schedule | CosineAnnealingLR, T_max = 4000, stepped every step |
| steps / batch | 4000 steps, batch 128 (gradient accumulation 4 x 32, same gradient) |
| precision | fp32 weights, TF32 matmuls |
| seed | 42 |
| augmentation | none (CLIPProcessor only) |
| data | the task's train split only; val and test are never trained on |
This is the default configuration of FusionBench's clip_finetune method (plus TF32 and gradient accumulation).
Tasks
The 20 FusionBench tasks and 80 further tasks (VTAB/ELEVATER, objects and scenes, fine-grained, food, agriculture,
medical, remote sensing, characters, documents, people, art, DomainNet). Each task has fixed train / val / test splits;
where the source publishes a split that is held out by group (object instance, video, slide, patient, hand model),
val is that held-out split, so val accuracy tracks test accuracy. RESULTS_clip-vit-large-patch14.md lists per-task
zero-shot and fine-tuned accuracy.
Licenses
These weights are derived from 100 public datasets with different licenses, some of them non-commercial (for example
CC BY-NC) and some without a stated license. The license of each source dataset applies to the checkpoint trained on it;
check the dataset's own terms before using a checkpoint. The base model is openai/clip-vit-large-patch14.
Loading
from transformers import CLIPModel, CLIPVisionModel
clip = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
vm = CLIPVisionModel.from_pretrained("bmmv-9x2q7/clip100-vit-large-patch14", subfolder="dtd/vision_model")
clip.vision_model.load_state_dict(vm.vision_model.state_dict())