Argus-Colqwen3.5-4b-v0 · bf16 release

Argus: Region-Aware Query-Conditioned Mixture of Experts for Visual Document Retrieval University of Innsbruck — Data Science group · 2026

DataScience-UIBK/Argus-Colqwen3.5-4b-v0-bf16 is the bfloat16 merged release of Argus-Colqwen3.5-4b. It is the exact same trained network as the fp32 sibling DataScience-UIBK/Argus-Colqwen3.5-4b-v0; only the LoRA-merge dtype differs.

The bf16 release is half the disk size (8.8 GB vs 18 GB), faster to download, and easier to deploy on memory-constrained GPUs — at the cost of ~0.1 pp on ViDoRe V1 and ~0.2 pp on V2 (merge-time precision loss).

Use this bf16 release if you want the smallest deployable artefact and can tolerate sub-percent score loss. Use the fp32 sibling if you need leaderboard-grade reproducibility or you're going to merge / quantise further downstream.

TL;DR — leaderboard standing

• #1 on the ViDoRe v1 leaderboard among 4B-class models (V1 = 0.9214) — beats Nemotron-4B-v2 (0.916), athrael-soju-colqwen3.5-4.5B (0.915), Ops-Colqwen3-4B (0.914).
• #2 overall on the ViDoRe v1 leaderboard, behind only the 8B Nemotron-vl-8b-v2 (0.927).
• Competitive on ViDoRe v2 (0.6380 nDCG@5) within the 4B class.
• 4 B parameters, 1024-d per-token embedding, ≤ 2048 visual tokens / page — fits on a single 12 GB GPU at bf16.

What is novel here

Most ColPali-style retrievers project every visual token through the same dense head, no matter what the query is. Argus replaces that dense head with a sparse mixture in which the gates depend on both the visual token and a pooled query summary, so the same page gets routed differently for different queries:

1. Region pooling — visual tokens are grouped into 4-token regions before routing.
2. Query-conditioned latent gating (GateScalars) — router input is region + region_coord_proj(coords) + query_context_proj(pooled_query). The query summary makes routing task-aware: a financial-numbers query routes through a different expert than a layout query, on the same page.
3. Sparse top-k=2 of 4 latent specialists, fused with an always-on shared dense expert via two learnable gating scalars.
4. Region-aware load balancing — load balance + KL-uniform + 0.01·router-z² aux losses suppress routing collapse.
5. 3-stage curriculum — dense baseline (teacher) → MoE balance warmup → joint retrieval with KL distillation.

Router sits at backbone layer −5.

Model details

Why two dtypes?

Merging a LoRA into the base requires materialising (α/r)·A·B and adding it to the base weight matrix.

• In bf16, both the delta cast and the addition lose precision — the gap is small but irreversible.
• In fp32, both are exact.

We publish both:

• fp32 preserves trained precision exactly. Larger disk, identical inference behaviour to the fp32 master used during evaluation.
• bf16 is the dtype most users will load anyway (torch_dtype=torch.bfloat16). Smaller disk, the merge precision loss is baked in.

If your downstream pipeline loads in bf16, the inference-time difference between the two repos is tiny — but the bf16 release saves bandwidth and disk. Use the fp32 sibling if you load in fp32 / fp64 (rare) or you're going to quantise further.

Performance — ViDoRe v1 (English, nDCG@5, 10 tasks)

Real per-task scores, measured with mteb 2.12 on the published bf16 weights.

🏆 = best in the 4B class for that task (cross-checked against published numbers from Ops-Colqwen3-4B, TomoroAI-colqwen3-embed-4b, SauerkrautLM-ColQwen3-4b, athrael-soju-colqwen3.5-4.5B).

ViDoRe v1 — 4B-class leaderboard comparison

(Only model surpassing Argus-4B on V1 overall is the 8B Nemotron-vl-8b-v2 at 0.927.)

Performance — ViDoRe v2 (English, nDCG@5, 4 tasks)

ViDoRe v2 — 4B-class context

V2 is the area we are still actively improving — Ops's wider 2560-d head pulls ahead on layout-heavy ESG / economics pages. Argus's per-token compression to 1024-d is a 3× storage saving over Ops at the cost of a small V2 gap; the V1 lead more than compensates for retrieval workloads dominated by document QA.

ViDoRe v3

Not yet evaluated for this release.

Storage cost

Installation

pip install "transformers>=5.0.0,<6.0.0"
pip install "mteb>=2.12,<3.0.0"
pip install -U "transformers>=5.0,<6.0"
pip install flash-attn==2.6.3 --no-build-isolation     # optional
rm -rf ~/.cache/huggingface/modules/transformers_modules

Usage

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

MODEL_ID = "DataScience-UIBK/Argus-Colqwen3.5-4b-v0-bf16"
DEVICE   = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,                  # this release ships in bf16
    attn_implementation="flash_attention_2",
    device_map=DEVICE,
).eval()

processor = AutoProcessor.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    max_num_visual_tokens=2048,
)

queries = [
    "What is the company's revenue in 2019?",
    "How does the proposed model compare to baselines?",
]
documents = [
    Image.open("page_a.png").convert("RGB"),
    Image.open("page_b.png").convert("RGB"),
]

q_emb  = model.encode_queries(processor, queries)
d_emb  = model.encode_images(processor, documents)
scores = processor.score(q_emb, d_emb)
print(scores)

Reproduce ViDoRe results with MTEB

import mteb

m  = mteb.get_model("DataScience-UIBK/Argus-Colqwen3.5-4b-v0-bf16")
v1 = mteb.get_benchmark("ViDoRe(v1)").tasks
v2 = mteb.get_benchmark("ViDoRe(v2)").tasks
mteb.MTEB(tasks=v1 + v2).run(m, encode_kwargs={"batch_size": 4})

Training

Same recipe as the fp32 sibling — see its card for full details. Only difference is the merge dtype.

Limitations

• English-dominant.
• Merge-time bf16 cast is irreversible — you cannot recover fp32 numbers by upcasting after load.
• ViDoRe v3 not yet evaluated.
• ~0.1 pp / ~0.2 pp behind the fp32 sibling on V1 / V2 — use that one if leaderboard parity matters.

License

Apache 2.0, inherited from Qwen3.5-VL-4B-Instruct.

Citation

@article{abdallah2026argus,
  title={Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval},
  author={Abdallah, Abdelrahman and Abdalla, Mahmoud and Ali, Mohammed and Jatowt, Adam},
  journal={arXiv preprint arXiv:2606.04300},
  year={2026}
}

Contact

• Org: DataScience-UIBK, University of Innsbruck
• Issues: open one on this repo's Community tab.