MoDA — LLaVA-MoRE 8B (CLIP ViT-L/14-336)
Official checkpoint for "MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs" (ICML 2026).
📄 Paper: hf.co/papers/2506.01850 · arXiv:2506.01850 💻 Code: github.com/waybarrios/MoDA 🌐 Project page: waybarrios.com/MoDA
MoDA (Modulation Adapter) is a lightweight module that improves fine-grained visual grounding in Multimodal LLMs through instruction-guided, channel-wise modulation of pre-aligned visual features. A stack of cross-attention layers conditions on the language instruction and produces a soft channel-wise mask (via sigmoid) that is applied multiplicatively (Hadamard product) to the aligned visual features before they reach the LLM:
Ṽ_aligned = V_aligned ⊙ σ(W · F(T, V_aligned))
MoDA adds only <1% FLOPs and 3.7% parameters, requires no extra data or supervision, and plugs into the standard two-stage LLaVA instruction-tuning pipeline.
This is the CLIP variant of LLaVA-MoRE 8B + MoDA. See MoDA-LLaVA-MoRE-8B-SigLIP-S2 for the SigLIP-S2 variant.
Model details
| Architecture | LLaVA-MoRE (LlavaLlamaForCausalLM) + MoDA adapter (2 cross-attention decoder layers, 16 heads — paper config) |
| LLM backbone | meta-llama/Llama-3.1-8B-Instruct |
| Vision encoder | openai/clip-vit-large-patch14-336 |
| Projector | 2-layer MLP (GELU) |
| Training data | LLaVA v1.5 instruction-tuning mix (665K) |
| Training recipe | Two-stage LLaVA protocol: (1) projector pretraining; (2) MoDA + LLM joint fine-tuning |
| Precision | bfloat16 |
Results
Selected results for this checkpoint (LLaVA-MoRE CLIP backbone; see the paper for the full 12-benchmark evaluation):
| Benchmark | Baseline | + MoDA |
|---|---|---|
| POPE | 85.1 | 86.3 |
| GQA | 63.6 | 64.4 |
| ScienceQA | 76.3 | 77.8 |
| LLaVA-Wild | 71.2 | 73.9 |
| RealWorldQA | 57.1 | 58.0 |
| MMVP | 27.3 | 28.7 |
| V*Bench | 42.8 | 44.0 |
Gains are strongest on fine-grained, vision-centric, and hallucination tasks.
Usage
This checkpoint uses the LLaVA-MoRE + MoDA codebase (not vanilla transformers):
git clone https://github.com/waybarrios/MoDA.git
cd MoDA/llava-more
pip install -r requirements.txt
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path
model_path = "waybarrios/MoDA-LLaVA-MoRE-8B-CLIP"
tokenizer, model, image_processor, context_len = load_pretrained_model(
model_path=model_path,
model_base=None,
model_name=get_model_name_from_path(model_path),
)
See the repository README for training instructions.
Evaluation
We evaluate with lmms-eval, which provides standardized implementations of all benchmarks reported in the paper (GQA, ScienceQA, POPE, MMBench, RealWorldQA, and more).
Citation
@inproceedings{barrios2026moda,
title = {MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs},
author = {Barrios, Wayner and Villa, Andr\'es and Leon Alcazar, Juan C. and Jin, SouYoung and Ghanem, Bernard},
booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},
year = {2026}
}
License
The MoDA code is released under the MIT License. This checkpoint is a derivative of Llama 3.1 and is distributed under the Llama 3.1 Community License.
Acknowledgments
Built on LLaVA and aimagelab/LLaVA-MORE. Supported by startup funds from Dartmouth College and by KAUST — Center of Excellence for Generative AI (award 5940).