Argus-Colqwen3.5-9b-v0 · bf16 release

Argus: Region-Aware Query-Conditioned Mixture of Experts for Visual Document Retrieval University of Innsbruck — Data Science group · 2026

DataScience-UIBK/Argus-Colqwen3.5-9b-v0-bf16 is the bfloat16 merged release of Argus-Colqwen3.5-9b. It is the exact same trained network as the fp32 sibling DataScience-UIBK/Argus-Colqwen3.5-9b-v0; only the LoRA-merge dtype differs.

The bf16 release is half the disk size (17 GB vs 33 GB), faster to download, and easier to deploy on memory-constrained GPUs — at the cost of ~0.001 pp on V1 and ~0.002 pp on V2 (well within ViDoRe eval noise).

Use this bf16 release if you want a smaller deployable artefact and can tolerate sub-percent score differences (almost every deployment scenario). Use the fp32 sibling if you need leaderboard-grade reproducibility or you're going to merge / quantise further downstream.

TL;DR — leaderboard standing

• Co-leads the ViDoRe v1 leaderboard at V1 = 0.9259 — within 0.001 of the 8B Nemotron-vl-8b-v2 (0.927) and ahead of every other public retriever.
• V2 = 0.6897 — best Argus result on V2, +0.05 over the 4B sibling.
• 8.8 B parameters, 1024-d per-token embedding, ≤ 2048 visual tokens / page — fits on a single 24 GB GPU at bf16 inference.

What is novel here

Most ColPali-style retrievers project every visual token through the same dense head, no matter what the query is. Argus replaces that dense head with a sparse mixture in which the gates depend on both the visual token and a pooled query summary, so the same page gets routed differently for different queries:

1. Region pooling — visual tokens are grouped into 4-token regions before routing.
2. Query-conditioned latent gating (GateScalars) — router input is region + region_coord_proj(coords) + query_context_proj(pooled_query). The query summary makes routing task-aware: a financial-numbers query routes through a different expert than a layout query, on the same page.
3. Sparse top-k=2 of 4 latent specialists, fused with an always-on shared dense expert via two learnable gating scalars.
4. Region-aware load balancing — load balance + KL-uniform + 0.01·router-z² aux losses suppress routing collapse.
5. 3-stage curriculum — dense baseline (teacher) → MoE balance warmup → joint retrieval with KL distillation. The 9B joint stage was extended on VDR1.5M + Docmatix (vdr_docmatix_full).

Router sits at backbone layer −5.

Model details

Why two dtypes?

Merging a LoRA into the base requires materialising (α/r)·A·B and adding it to the base weight matrix.

• In bf16, both the delta cast and the addition lose precision — the gap is small but irreversible.
• In fp32, both are exact.

For the 9B release the merge cost is ~0.001 V1 / ~0.002 V2 — within the noise floor of a single ViDoRe run. We publish both:

• fp32 preserves trained precision exactly. Larger disk, identical inference behaviour to the fp32 master used during evaluation.
• bf16 is the dtype most users will load anyway. Half the disk, the merge precision loss is baked in but immaterial in practice.

If your downstream pipeline loads in bf16, the bf16 release is the strict win — same scores within rounding, half the disk.

Performance — ViDoRe v1 (English, nDCG@5, 10 tasks)

Real per-task scores, measured with mteb 2.12 on the published bf16 weights, side-by-side with every Argus sibling.

ViDoRe v1 — overall leaderboard comparison

Performance — ViDoRe v2 (English, nDCG@5, 4 tasks)

ViDoRe v2 — overall context

ViDoRe v3

Not yet evaluated for this release.

Storage cost

Installation

pip install "transformers>=5.0.0,<6.0.0"
pip install "mteb>=2.12,<3.0.0"
pip install -U "transformers>=5.0,<6.0"
pip install flash-attn==2.6.3 --no-build-isolation     # optional
rm -rf ~/.cache/huggingface/modules/transformers_modules

Usage

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

MODEL_ID = "DataScience-UIBK/Argus-Colqwen3.5-9b-v0-bf16"
DEVICE   = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,                  # this release ships in bf16
    attn_implementation="flash_attention_2",
    device_map=DEVICE,
).eval()

processor = AutoProcessor.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    max_num_visual_tokens=2048,
)

queries = [
    "What is the company's revenue in 2019?",
    "How does the proposed model compare to baselines?",
]
documents = [
    Image.open("page_a.png").convert("RGB"),
    Image.open("page_b.png").convert("RGB"),
]

q_emb  = model.encode_queries(processor, queries)
d_emb  = model.encode_images(processor, documents)
scores = processor.score(q_emb, d_emb)
print(scores)

Reproduce ViDoRe results with MTEB

import mteb

m  = mteb.get_model("DataScience-UIBK/Argus-Colqwen3.5-9b-v0-bf16")
v1 = mteb.get_benchmark("ViDoRe(v1)").tasks
v2 = mteb.get_benchmark("ViDoRe(v2)").tasks
mteb.MTEB(tasks=v1 + v2).run(m, encode_kwargs={"batch_size": 2})

Training

Same recipe as the fp32 sibling — see its card for full details. Only difference is the merge dtype.

When to use which Argus variant

Limitations

• English-dominant.
• Merge-time bf16 cast is irreversible — you cannot recover fp32 numbers by upcasting after load.
• ViDoRe v3 not yet evaluated.
• Needs ~17 GB VRAM at bf16 inference; single-GPU users need ≥ 24 GB.

License

Apache 2.0, inherited from Qwen3.5-VL-9B-Instruct.

Citation

@article{abdallah2026argus,
  title={Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval},
  author={Abdallah, Abdelrahman and Abdalla, Mahmoud and Ali, Mohammed and Jatowt, Adam},
  journal={arXiv preprint arXiv:2606.04300},
  year={2026}
}

Contact

• Org: DataScience-UIBK, University of Innsbruck
• Issues: open one on this repo's Community tab.