SAM 3 × Qwen3.5-2B

Qwen3.5-2B with SAM 3's Perception Encoder trunk fused into its visual tokens.

SAM 3's backbone is contrastively pretrained and detection-tuned, which makes it strong at exactly what a small VLM is weakest at: precise localization. It is added alongside Qwen's native vision tower rather than replacing it, so none of Qwen's pretrained alignment is discarded.

How it works

                 ┌─ Qwen ViT ──────────> (N, 2048) ────────────────┬──> (+) ──> LLM
image ──┤                                  grid (h/2, w/2)         │     ▲
                 └─ SAM 3 ViT ─────────> (72, 72, 1024)            │     │
                        └─ interpolate ─> (N, 1024) ───────────────┘     │
                           concat(3072) ─> MLP ─> (N, 2048) ─> * gate ───┘

fused = qwen + gate * MLP([qwen ; sam3_resampled]), where gate is a learned per-channel parameter initialised to zero — so before training the model is bit-identical to stock Qwen3.5-2B, and training can only add signal.

SAM 3's dense feature map is resampled onto Qwen's post-merge token grid and folded in channel-wise. Token counts, image_grid_thw, mRoPE, the processor and the chat template are all untouched: the entire integration is a wrapper around get_image_features.

The fused model was tuned for object grounding on a private dataset that is not released, so it is specialised rather than general-purpose — see Limitations.

Inputs

Takes exactly the same inputs as stock Qwen3.5-2B. SAM 3's 1008×1008 input is reconstructed inside the model from the pixel_values the standard processor already produces, so no custom processor or extra image input is needed.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model = AutoModelForImageTextToText.from_pretrained(
    "<repo-id>", dtype=torch.bfloat16, trust_remote_code=True, device_map="cuda"
).eval()
processor = AutoProcessor.from_pretrained(
    "<repo-id>", trust_remote_code=True, max_pixels=802816
)

image = Image.open("photo.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Detect every <object> in this image. Return each as JSON with a bbox_2d field."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
batch = processor(text=[text], images=[image], return_tensors="pt").to("cuda")

out = model.generate(**batch, max_new_tokens=256, do_sample=False)
print(processor.tokenizer.decode(out[0][batch["input_ids"].shape[1]:], skip_special_tokens=True))

Output is JSON with coordinates normalised to 0–1000 against the original image size — Qwen3.5's native grounding convention:

[{"bbox_2d": [741, 320, 791, 448], "label": "..."}]

Ablation toggle

The SAM 3 branch can be switched off at runtime, so one checkpoint gives both the fused model and its Qwen-only counterpart:

model.sam3_enabled = False   # Qwen-only variant
model.sam3_enabled = True    # fused (default)

Serving with vLLM

vLLM implements model architectures itself and does not run a repo's remote modeling code, so this architecture needs the small out-of-tree plugin shipped in vllm_plugin/ inside this repo. Installing it registers Sam3Qwen3_5ForConditionalGeneration via a vllm.general_plugins entry point, which fires in every vLLM process — including the EngineCore subprocess, where a manual register() call would not reach.

huggingface-cli download <repo-id> --local-dir sam3-qwen3.5-2b
pip install -e sam3-qwen3.5-2b/vllm_plugin

vllm serve sam3-qwen3.5-2b \
  --served-model-name sam3-qwen3.5-2b \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --mm-processor-kwargs '{"max_pixels": 802816, "min_pixels": 65536}'

--mm-processor-kwargs matters: the bundled processor keeps Qwen3.5's stock 16.7M-pixel ceiling (4096²), which balloons sequence length on large images. All reported numbers use the 802816 cap above.

Query it through the OpenAI-compatible API:

IMG=$(base64 -w0 photo.jpg)
curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"sam3-qwen3.5-2b\",\"temperature\":0,\"max_tokens\":256,
       \"messages\":[{\"role\":\"user\",\"content\":[
         {\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/jpeg;base64,$IMG\"}},
         {\"type\":\"text\",\"text\":\"Detect every <object> in this image. Return each as JSON with a bbox_2d field.\"}]}]}"

Or with the Python API, registering the plugin before constructing LLM:

import vllm_sam3_qwen; vllm_sam3_qwen.register()
from vllm import LLM, SamplingParams

llm = LLM(model="sam3-qwen3.5-2b", trust_remote_code=True, dtype="bfloat16",
          mm_processor_kwargs={"max_pixels": 802816, "min_pixels": 65536})

Throughput on one RTX A6000, vLLM 0.25.1, batched, single image per request:

SAM 3's extra 446M-parameter encoder pass costs roughly 2× serving throughput. Whether that is worth it depends on how much your task values localization precision.

Limitations

• Specialised, not general-purpose. Tuned for object grounding on a single private dataset with one prompt phrasing. Behaviour on other domains, other phrasings, and on general VQA is untested and may be worse than stock Qwen3.5-2B.
• Precision-biased. The fusion mainly suppresses false positives; it does not make the model find more objects. Recall is not improved.
• Still images only — video input raises NotImplementedError.
• SAM 3 runs at a fixed 1008×1008 regardless of input resolution, and its input side must be a multiple of 336 (patch 14 × window 24).
• bfloat16 only has been validated.

Licence

This checkpoint bundles weights from two sources with different licences:

• Qwen3.5-2B — Apache-2.0.
• SAM 3 (facebook/sam3) — a custom Meta licence. The upstream repo is gated with manual, per-user approval.

The SAM 3 licence therefore governs use of this model. Review the terms at https://huggingface.co/facebook/sam3 before using or redistributing these weights.