Fashion Florence: Structured Fashion Tagging with Florence-2

A fine-tuned Florence-2-large model that analyzes clothing images and returns structured fashion tags as JSON.

Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks. This fine-tuned version specializes in structured fashion attribute extraction.

Model Summary

Output Schema

Given a clothing image, the model outputs structured JSON with:

Evaluation Results

Evaluated on 3,915 held-out test examples (5% split from iMaterialist).

Overall Metrics

Per-Category Performance

Color and material use exact-match scoring; near-synonyms (e.g. "navy" vs "navy blue") count as misses. Shoes and accessory have low support in the test set.

Comparison with Other Approaches

While direct comparisons are approximate (different datasets and task setups), these benchmarks provide context:

Fashion Florence outperforms zero-shot LLMs by a wide margin on fashion attribute prediction, and approaches specialized classifiers on category accuracy while also producing full structured output.

How to Get Started with the Model

import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    "anushreeberlia/fashion-florence",
    torch_dtype=torch_dtype,
    trust_remote_code=True,
).to(device)
processor = AutoProcessor.from_pretrained(
    "anushreeberlia/fashion-florence", trust_remote_code=True,
)

image = Image.open("clothing_image.jpg").convert("RGB")
prompt = "Analyze this clothing item image and return structured fashion tags as JSON."

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=256,
    num_beams=3,
)
result = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(result)

Training Details

• Epochs: 2
• Learning rate: 2e-4
• Effective batch size: 16 (1 × 16 gradient accumulation)
• LoRA rank: 16, alpha: 32, dropout: 0.05
• Target modules: all linear layers
• Precision: fp16
• Dataset: ~78k training examples from iMaterialist

Citation

@article{xiao2023florence,
  title={Florence-2: Advancing a unified representation for a variety of vision tasks},
  author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
  journal={arXiv preprint arXiv:2311.06242},
  year={2023}
}