ColQwen3.5-4.5B-v3

A visual document retrieval model using ColBERT-style late interaction with Qwen3.5-4B.

4.5B parameters | 128-dim embeddings | LoRA (r=16, alpha=64) | BF16

Note on embed dim. Projection head is 320-dim (trained). config.json sets dim: 320 (read by ColQwen3_5) with embed_dim: 320 as a synonym for cross-ecosystem tools, so ColQwen3_5.from_pretrained(...) loads correctly by default. Leaderboard columns showing 128 reflect the colpali-engine class default (self.dim = getattr(config, "dim", 128)) and not the actual output dim.

Design Philosophy

V3 builds on V2 with automated hyperparameter search and evolutionary model soup merging. The search identified an optimal LoRA config (r=16, alpha=64) with cosine scheduling, and the final checkpoint is a per-layer evolutionary merge with V2. The result is a +0.0219 improvement over V2 across benchmarks.

Benchmark Results

ViDoRe V3 (nDCG@10)

Rankings on the live ViDoRe V3 leaderboard drift as new submissions land. At release, v3 held #3 overall; figures below were captured on 2026-04-20, where v3 sits at #6 on Mean (Task) and remains top-3 among 4B-class models.

ViDoRe V1+V2 (nDCG@5)

Limitations

• Trails Nemotron 8B on most V3 tasks (1.9 points gap on Mean Task)
• V1+V2 average slightly below Ops-ColQwen3 and Nemotron variants
• V2 is the weakest benchmark area (ESG, Econ)
• Larger 4B+ submissions (webAI-ColVec1-4b/9b) released after v3 now lead overall Mean (Task); v3 remains top-3 among 4B-class models

Usage

[!NOTE] This model requires transformers>=5.2.0. If you built an index with an older setup, re-encode your documents: older versions ran the model with causal attention, so the embeddings differ.

Sentence Transformers

This model can be used with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:

pip install "sentence-transformers[image]>=6.0.0"

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("athrael-soju/colqwen3.5-4.5B-v3")

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
images = [
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
]

query_embeddings = model.encode_query(queries)
image_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, image_embeddings[0].shape)
# torch.Size([23, 320]) torch.Size([755, 320])

# Diagonal should have higher scores
scores = model.similarity(query_embeddings, image_embeddings)
print(scores)
# tensor([[17.4434, 10.4502],
#         [ 5.0303, 13.8125]], device='cuda:0')

ColPali Engine

import torch
from PIL import Image
from colpali_engine.models import ColQwen3_5, ColQwen3_5Processor

model = ColQwen3_5.from_pretrained(
    "athrael-soju/colqwen3.5-4.5B-v3",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    attn_implementation="sdpa",
)
processor = ColQwen3_5Processor.from_pretrained("athrael-soju/colqwen3.5-4.5B-v3")

# Embed document images
images = [Image.open("page1.png"), Image.open("page2.png")]
batch = processor.process_images(images).to(model.device)
with torch.no_grad():
    doc_embeddings = model(**batch)

# Embed queries
queries = ["What is the revenue for Q4?", "Show me the organizational chart"]
batch = processor.process_queries(queries).to(model.device)
with torch.no_grad():
    model.rope_deltas = None
    query_embeddings = model(**batch)

# Score with MaxSim
scores = processor.score(query_embeddings, doc_embeddings)

Training

Pipeline

1. Hyperparameter search: multi-objective optimization across V1+V3, found optimal LoRA config
2. Full training: 3 seeds (42, 123, 456) with optimized hyperparameters, 1 epoch
3. Seed merge: full state dict averaging (3 seeds into 1)
4. Model soup: per-layer evolutionary merge with V2

Training Data (~776K pairs)

• vidore/colpali_train_set: 127K
• openbmb/VisRAG-Ret-Train-Synthetic-data: 239K
• openbmb/VisRAG-Ret-Train-In-domain-data: 123K
• llamaindex/vdr-multilingual-train: ~270K (5 languages)
• vidore/tatdqa_train: ~13K (finance)
• Metric-AI/tabfquad_train_set: ~1.5K (tables)

Hyperparameters

Technical Notes

• PEFT's add_weighted_adapter is broken for ColQwen3.5 (both DARE-TIES and linear). Use full state dict averaging for seed merging.
• Model soup done via direct state dict interpolation with per-layer optimized weights.
• B200/Blackwell GPUs require Conv3d to F.linear monkey-patch.
• Always clear rope_deltas before forward passes with hard negatives.

Transparency

The complete evaluation trail from V1, V2, and V3 development is available at athrael-soju/colqwen-optimization-trail. This includes every intermediate evaluation showing which candidates were tried, what scores they got, and which were selected for publication. All selection decisions were evaluated against the same public ViDoRe benchmarks used for final reporting.

Archived Versions (V1 and V2)

V1 and V2 model snapshots have been consolidated into this repository to reduce sprawl and preserve provenance. Full contents of the previously-separate colqwen3.5-4.5B-v1 and colqwen3.5-4.5B-v2 repositories are available here under:

• v1/ — complete V1 snapshot (weights, model card, tokenizer, configs)
• v2/ — complete V2 snapshot (weights, model card, tokenizer, configs)

Each subfolder contains the full original file set (model.safetensors, README.md, config.json, chat_template.jinja, processor_config.json, tokenizer.json, tokenizer_config.json).

To load an archived version:

from colpali_engine.models import ColQwen3_5, ColQwen3_5Processor
# V1
model = ColQwen3_5.from_pretrained("athrael-soju/colqwen3.5-4.5B-v3", subfolder="v1", ...)
processor = ColQwen3_5Processor.from_pretrained("athrael-soju/colqwen3.5-4.5B-v3", subfolder="v1")
# V2
model = ColQwen3_5.from_pretrained("athrael-soju/colqwen3.5-4.5B-v3", subfolder="v2", ...)

The top-level files remain the V3 model — the recommended version for production use.

Citation

@misc{colqwen35v3,
  title={ColQwen3.5-v3: Visual Document Retrieval with Evolutionary Model Soups},
  author={athrael-soju},
  year={2026},
  url={https://huggingface.co/athrael-soju/colqwen3.5-4.5B-v3}
}

License

Apache 2.0