Fashion Image Search
Visual product search: a photo goes in, the most similar products of a fashion catalog come out. A frozen ImageNet MobileNetV2 turns each image into a 1280-d feature vector, PCA (fit on the catalog, no labels) reduces it to a 256-d L2-normalized embedding, and an exact cosine nearest-neighbour search over the pre-computed embeddings of 8,000 catalog products returns the closest matches with their name, category and a thumbnail. Nothing is fine-tuned: "training" means embedding the catalog and measuring retrieval quality on held-out photos.
On 500 held-out test photos, 72.8 % of the top-5 matches have the query's exact article type (119 types in the catalog; a random ranking gets 5.4 %) and 97.0 % its master category.
Model
| Embedder | keras.applications.MobileNetV2(include_top=False, pooling="avg", weights="imagenet"), frozen, 2,257,984 parameters |
| Preprocessing | RGB, bilinear resize to 224x224, x / 127.5 - 1 (a Rescaling layer inside backbone.keras) |
| Embedding | 1280-d global-average-pooled features -> PCA 256-d (projection.npz, fit on the catalog embeddings) -> L2-normalized |
| Index | 8,000 catalog products, embeddings.npy (8000x256 float16) + catalog.parquet (metadata + JPEG thumbnails, native 60x80 px) |
| Search | exact cosine k-NN (brute force; same ranking as sklearn NearestNeighbors(metric="cosine")) |
| Input / output | one image -> {"matches": [{"id", "name", "master_category", "article_type", "score", "thumbnail"}]} |
Files: backbone.keras (the embedder, so inference never downloads ImageNet weights),
projection.npz, embeddings.npy, catalog.parquet, config.json, model.py (embedder, preprocessing, search,
Predictor), handler.py (Inference Endpoint), metrics.json, assets/ (graphs).
Usage
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("shalev396/fashion-image-search")
sys.path.insert(0, path)
import model
predictor = model.load(path, device="cpu")
result = predictor.predict("my_sneaker.jpg", k=6) # PIL image, path, bytes or base64
for m in result["matches"]:
print(f'{m["score"]:.3f} {m["article_type"]:12s} {m["name"]}')
- Space API (free): see the Space:
POST /gradio_api/call/predictwith[image, k]->[{"matches": [...]}, seconds, device]. - Inference Endpoint: deploy this repo (Deploy -> Inference Endpoints).
handler.pytakes{"inputs": "<base64 image>", "parameters": {"k": 6}}and returns the same{"matches": [...]}.
Training
Data: Fashion Product Images (Small) (44k Myntra product photos, 60x80 px, labelled with master category / sub category / article type). Master categories with fewer than 20 products are dropped. A stratified (by master category) split gives the catalog (8,000 products that get indexed), 500 validation queries (choose the variant) and 500 test queries (reported once). Queries are never in the catalog.
Five representations are compared with the same cosine k-NN search: raw pixels, a colour histogram, MobileNetV2 and MobileNetV2 reduced with PCA (fit on the catalog, no labels). The one with the best validation Precision@10 (article type) is exported. Embedding the ~9,000 photos took 767 s on a shared (heavily loaded) desktop CPU, 879 s for the whole build; an earlier run of the same pipeline on an idle machine took 256 s end to end. Full code: training/ · Colab
Experiments
| variant | dim | val P@10 (article type) | test P@5 (article type) | test P@10 (article type) | test P@5 (master cat.) | test P@10 (master cat.) |
|---|---|---|---|---|---|---|
| mobilenetv2_pca256 (deployed) | 256 | 0.7090 | 0.7284 | 0.7022 | 0.9704 | 0.9686 |
| mobilenetv2_pca128 | 128 | 0.7084 | 0.7252 | 0.7026 | 0.9720 | 0.9696 |
| mobilenetv2 | 1,280 | 0.7038 | 0.7264 | 0.7012 | 0.9680 | 0.9650 |
| pixels_32 | 3,072 | 0.5988 | 0.6392 | 0.6066 | 0.9384 | 0.9292 |
| color_hist_8 | 512 | 0.2988 | 0.2868 | 0.2614 | 0.6904 | 0.6656 |
The three MobileNetV2 variants are within 0.005 of each other on validation (500 queries, so the differences are within noise); PCA-256 was picked by the rule and keeps the index 5x smaller than the full 1280-d embedding. The CNN clearly beats both no-learning baselines, especially on the fine-grained article type (P@10 0.70 vs 0.61 for raw pixels and 0.26 for colour histograms).
Evaluation
Precision@K = share of a held-out query's top-K catalog matches that have the same label (master category: 6 coarse classes; article type: 119 fine classes in the catalog, such as Tshirts, Casual Shoes, Handbags). A random ranking scores 0.342 (master category) and 0.054 (article type). Test split of the deployed variant:
| metric (test) | value |
|---|---|
| p_at_1_master_category | 0.9780 |
| p_at_5_master_category | 0.9704 |
| p_at_10_master_category | 0.9686 |
| p_at_1_article_type | 0.7520 |
| p_at_5_article_type | 0.7284 |
| p_at_10_article_type | 0.7022 |
Per master category (test P@5, article type; metrics.json -> per_category, n = test queries): Apparel 0.74 (n=241), Accessories 0.82 (n=127), Footwear 0.63 (n=103), Personal Care 0.57 (n=27), Free Items 1.00 (n=1), Sporting Goods 0.40 (n=1).
Footwear mixes up visually close types (heels / flats / sandals). An earlier run of the legacy code (full 1280-d embedding, a different 500-query sample) reported P@5 0.977 / 0.718 (master category / article type), in line with these numbers.
Limitations
- The catalog is fixed: 8,000 products from one Indian e-commerce site (Myntra), photographed on a white background. Street photos, several items in one picture or unusual angles match less well.
- ImageNet features know shapes and textures, not fashion attributes: two shirts of the same cut but different brands look alike, and "similar" means visually similar, not same style or price.
- Precision@K uses the dataset's labels as ground truth. Some near-duplicates carry different article-type labels (e.g. Tshirts vs Tops), so the fine-grained number understates quality a bit.
- Products photographed in their packaging match other boxed products: the watch-in-a-box example (second row of the query grid) retrieves fragrance gift sets, not watches.
- Rare master categories are barely represented (Free Items 19, Sporting Goods 4 catalog products).
- The input photo is squashed to 224x224 without cropping; very wide or tall images are distorted.