MoDA — Qwen3-VL-2B
Official checkpoint for "MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs" (ICML 2026).
📄 Paper: hf.co/papers/2506.01850 · arXiv:2506.01850 💻 Code: github.com/waybarrios/MoDA 🌐 Project page: waybarrios.com/MoDA
MoDA (Modulation Adapter) is a lightweight module that improves fine-grained visual grounding in Multimodal LLMs through instruction-guided, channel-wise modulation of pre-aligned visual features. A stack of cross-attention layers conditions on the language instruction and produces a soft channel-wise mask (via sigmoid) that is applied multiplicatively (Hadamard product) to the aligned visual features before they reach the LLM:
Ṽ_aligned = V_aligned ⊙ σ(W · F(T, V_aligned))
This checkpoint applies MoDA to Qwen3-VL-2B-Instruct, showing that the gains generalize beyond CLIP-style encoders to modern MLLMs. Other variants: MoDA-LLaVA-MoRE-8B-SigLIP-S2 · MoDA-LLaVA-MoRE-8B-CLIP.
Model details
| Architecture | Qwen3-VL-2B + MoDA adapter (2 cross-attention decoder layers, 16 heads — paper config; 75.6M adapter params) |
| Base model | Qwen/Qwen3-VL-2B-Instruct |
| Training data | LLaVA v1.5 instruction-tuning mix (665K) |
| Precision | bfloat16 |
Results
Qwen3-VL-2B-Instruct baseline vs. + MoDA (see the paper for the full evaluation):
| Benchmark | Baseline | + MoDA |
|---|---|---|
| ScienceQA | 79.3 | 84.2 (+4.9) |
| RealWorldQA | 64.7 | 68.8 (+4.1) |
| GQA | 59.4 | 63.2 (+3.8) |
| MMStar | 53.9 | 55.3 |
| MMBench | 86.5 | 87.4 |
| POPE | 89.4 | 89.8 |
| CV-Bench | 80.9 | 81.0 |
Usage
Loads directly with 🤗 transformers — the MoDA classes ship inside this repo (trust_remote_code=True):
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
model_id = "waybarrios/MoDA-Qwen3-VL-2B"
model = AutoModelForImageTextToText.from_pretrained(
model_id, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)
messages = [
{"role": "user", "content": [
{"type": "image", "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
{"type": "text", "text": "Describe this image."},
]},
]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt"
).to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
Training and evaluation code (lmms-eval integration) lives in the qwen3-vl/ directory of the repository.
Evaluation
We evaluate with lmms-eval, which provides standardized implementations of all benchmarks reported in the paper (GQA, ScienceQA, POPE, MMBench, RealWorldQA, and more).
Citation
@inproceedings{barrios2026moda,
title = {MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs},
author = {Barrios, Wayner and Villa, Andr\'es and Leon Alcazar, Juan C. and Jin, SouYoung and Ghanem, Bernard},
booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},
year = {2026}
}
License
Apache-2.0, following the base model Qwen/Qwen3-VL-2B-Instruct.
Acknowledgments
Built on Qwen3-VL by Alibaba. Supported by startup funds from Dartmouth College and by KAUST — Center of Excellence for Generative AI (award 5940).