CLIP ViT-B/32 LoRA fine-tuned on Fashion Product Images
A LoRA fine-tune of openai/clip-vit-base-patch32's vision encoder, trained to improve
text-to-image retrieval on the Fashion Product Images
(Small) catalog
(44,412 products). Built as the backbone of a multimodal semantic search + zero-shot tagging
project. Full code, tests, and write-up: GitHub
repo.
What's in this repo
| Path | Contents |
|---|---|
clip_merged_fp16/ | The recommended way to use this model: base CLIP + LoRA merged into one checkpoint, saved in native fp16. Load directly with CLIPModel.from_pretrained(...), no peft needed. |
lora_adapter/ | The LoRA adapter alone (rank 16, vision encoder q_proj/k_proj/v_proj only, text encoder untouched), for use with peft.PeftModel.from_pretrained on top of the base fp32 checkpoint. |
embeddings_pretrained.npy / embeddings_lora.npy | Precomputed CLIP embeddings for all 44,412 catalog images, before and after fine-tuning, aligned to item_ids.npy. |
styles_clean.csv | Cleaned product metadata (id, product name, article type, color) matching the embeddings. |
eval_results.json | Recall@1/5/10 and MRR, pretrained vs. fine-tuned, on 40 held-out color+category query phrases never seen during training. |
split.json, loss_history.json | Train/eval id split and per-step training loss, for reproducibility. |
How to use (recommended: merged fp16 checkpoint)
from transformers import CLIPModel, CLIPProcessor
model = CLIPModel.from_pretrained("sidmhuggingface/clip-lora-fashion-search", subfolder="clip_merged_fp16")
processor = CLIPProcessor.from_pretrained("sidmhuggingface/clip-lora-fashion-search", subfolder="clip_merged_fp16")
Training details
- Base model:
openai/clip-vit-base-patch32 - Method: LoRA (rank 16, alpha 32, dropout 0.1) applied only to the vision encoder's
q_proj/k_proj/v_projattention projections. Text encoder frozen entirely. - Objective: CLIP contrastive loss (InfoNCE, in-batch negatives) on (image,
productDisplayName) pairs. - Data: 44,412 product images, 90% train / 10% id-level holdout (seed 42).
- Hardware: ASU Sol HPC, 1x NVIDIA A100-SXM4-80GB, 2 epochs, well under 10 minutes of GPU compute.
Evaluation
Measured on 40 held-out (color, article type) query phrases (e.g. "black formal shoes") never seen during fine-tuning, each checked against the full 44,412-item catalog using a from-scratch Recall@k / MRR implementation.
| Model | Recall@1 | Recall@5 | Recall@10 | MRR |
|---|---|---|---|---|
| Pretrained CLIP | 1.4% | 7.6% | 14.0% | 0.704 |
| LoRA fine-tuned | 1.6% | 9.1% | 17.2% | 0.745 |
Recall@10 improved +23.0% relative to the pretrained baseline. Absolute Recall@10 looks modest because these queries have large relevant sets (25-107 relevant images per query out of 44,412) — MRR (whether the first correct hit ranks early) is the more forgiving and arguably more representative metric for this catalog's query style, and improved +5.8% relative.
Limitations
CLIP has known weaknesses on compositional attribute binding — e.g. a query like "red dress" can return visually similar red garments (nightdresses, camisoles) rather than dresses specifically, consistent with findings in Thrush et al., 2022 (Winoground). This fine-tune improves retrieval quality on this catalog's query distribution but does not fix that underlying limitation.
License
MIT, matching the base openai/clip-vit-base-patch32 model.