MoDA — LLaVA-MoRE 8B (SigLIP-SO400M + S2)
Official checkpoint for "MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs" (ICML 2026).
📄 Paper: hf.co/papers/2506.01850 · arXiv:2506.01850 💻 Code: github.com/waybarrios/MoDA 🌐 Project page: waybarrios.com/MoDA
MoDA (Modulation Adapter) is a lightweight module that improves fine-grained visual grounding in Multimodal LLMs through instruction-guided, channel-wise modulation of pre-aligned visual features. A stack of cross-attention layers conditions on the language instruction and produces a soft channel-wise mask (via sigmoid) that is applied multiplicatively (Hadamard product) to the aligned visual features before they reach the LLM:
Ṽ_aligned = V_aligned ⊙ σ(W · F(T, V_aligned))
MoDA adds only <1% FLOPs and 3.7% parameters, requires no extra data or supervision, and plugs into the standard two-stage LLaVA instruction-tuning pipeline.
Model details
| Architecture | LLaVA-MoRE (LlavaLlamaForCausalLM) + MoDA adapter (2 cross-attention decoder layers, 16 heads — paper config) |
| LLM backbone | meta-llama/Llama-3.1-8B-Instruct |
| Vision encoder | google/siglip-so400m-patch14-384 with S2 multi-scale features (384 / 768 / 1152) |
| Projector | 2-layer MLP (GELU) |
| Training data | LLaVA v1.5 instruction-tuning mix (665K) |
| Training recipe | Two-stage LLaVA protocol: (1) projector pretraining; (2) MoDA + LLM joint fine-tuning |
| Precision | bfloat16 |
Results
Selected results for this checkpoint (LLaVA-MoRE SigLIP-S2 backbone; see the paper for the full 12-benchmark evaluation):
| Benchmark | Baseline | + MoDA |
|---|---|---|
| ScienceQA | 77.1 | 81.9 (+4.8) |
| POPE | 86.0 | 87.7 |
Gains are strongest on fine-grained, vision-centric, and hallucination tasks.
Usage
This checkpoint uses the LLaVA-MoRE + MoDA codebase (not vanilla transformers):
git clone https://github.com/waybarrios/MoDA.git
cd MoDA/llava-more
pip install -r requirements.txt
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path
model_path = "waybarrios/MoDA-LLaVA-MoRE-8B-SigLIP-S2"
tokenizer, model, image_processor, context_len = load_pretrained_model(
model_path=model_path,
model_base=None,
model_name=get_model_name_from_path(model_path),
)
See the repository README for training instructions.
Evaluation
We evaluate with lmms-eval, which provides standardized implementations of all benchmarks reported in the paper (GQA, ScienceQA, POPE, MMBench, RealWorldQA, and more).
Citation
@inproceedings{barrios2026moda,
title = {MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs},
author = {Barrios, Wayner and Villa, Andr\'es and Leon Alcazar, Juan C. and Jin, SouYoung and Ghanem, Bernard},
booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},
year = {2026}
}
License
The MoDA code is released under the MIT License. This checkpoint is a derivative of Llama 3.1 and is distributed under the Llama 3.1 Community License.
Acknowledgments
Built on LLaVA and aimagelab/LLaVA-MORE. Supported by startup funds from Dartmouth College and by KAUST — Center of Excellence for Generative AI (award 5940).