ColUltraVBERT
ColUltraVBERT is a 255M-parameter late-interaction retriever for visual documents, fine-tuned from UltraVBERT. It encodes every query token and every page token as a 768-d vector and scores a page against a query with MaxSim. The first 128, 256, 320 or 512 dimensions of each vector form a smaller index. Try it in the ColUltraVBERT Space.
Models
Data
Usage
Requires sentence-transformers>=6.1 and transformers>=5.20.
from PIL import Image
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("athrael-soju/ColUltraVBERT", trust_remote_code=True)
pages = [Image.open("page1.png"), Image.open("page2.png")]
queries = ["What was the 2023 revenue?", "Which table lists regional margins?"]
page_embeddings = model.encode_document(pages)
query_embeddings = model.encode_query(queries)
# One row per query, one column per page.
scores = model.similarity(query_embeddings, page_embeddings)
For a smaller index, keep the first 128, 256, 320 or 512 dimensions of every vector and L2-normalize them again; the tables below score each size.
Model
A learned linear layer maps each 768-d token state from UltraVBERT to a 768-d retrieval vector, trained with Matryoshka prefixes so the first 128, 256, 320 and 512 dimensions also work on their own. Each token vector is L2-normalized and padding tokens are dropped. The image processor resizes a page to a 3,072 px longest edge and splits it into up to 27 tiles of 512 px plus a downscaled copy of the whole page, so a page yields at most 1,792 image-token vectors.
Training
Four fine-tunes of UltraVBERT on VDR Train 500k, differing in hard negatives, distillation and seed, are merged layer by layer into the released weights. Each fine-tune trains LoRA adapters (rank 32) on the text encoder and the output projection for one epoch at batch 256; the vision encoder and connector stay frozen and shared.
ViDoRe
Scores from MTEB 2.26.3, in bf16 with SDPA attention, at 768 dimensions. A task with several language splits reports their mean. ColModernVBERT's scores are its published MTEB results.
| ViDoRe V1 (nDCG@5) | ColModernVBERT | ColUltraVBERT |
|---|---|---|
| ArxivQA | 73.2 | 80.3 |
| DocVQA | 55.6 | 57.7 |
| InfoVQA | 86.4 | 89.5 |
| TabFQuAD | 74.8 | 92.5 |
| TAT-DQA | 73.1 | 75.1 |
| ShiftProject | 60.4 | 81.3 |
| SyntheticDocQA AI | 97.9 | 99.6 |
| SyntheticDocQA Energy | 93.3 | 95.3 |
| SyntheticDocQA Government Reports | 92.3 | 95.1 |
| SyntheticDocQA Healthcare Industry | 95.3 | 97.9 |
| Mean | 80.2 | 86.4 |
| ViDoRe V2 (nDCG@5) | ColModernVBERT | ColUltraVBERT |
|---|---|---|
| ESG Reports | 40.1 | 49.7 |
| Economics Reports | 29.2 | 51.5 |
| BioMedical Lectures | 31.1 | 54.3 |
| ESG Reports HL | 52.2 | 63.6 |
| Mean | 38.2 | 54.8 |
| ViDoRe V3 (nDCG@10) | ColModernVBERT | ColUltraVBERT |
|---|---|---|
| Computer Science | 37.6 | 66.5 |
| Energy | 32.5 | 51.9 |
| Finance EN | 28.8 | 53.4 |
| Finance FR | 19.8 | 33.3 |
| HR | 19.8 | 51.2 |
| Industrial | 15.7 | 37.7 |
| Pharmaceuticals | 32.2 | 57.6 |
| Physics | 22.6 | 37.9 |
| Mean | 26.1 | 48.7 |
Smaller indexes
Means at each vector size and page budget over the same ViDoRe tasks, scored on cached page features, which agree
with MTEB within 0.0035 nDCG per task. The 2,048 px, 16-tile budget sets size.longest_edge to 2048 and
max_image_tiles to 16 in processor_config.json and caps a page at 1,088 vectors.
| Vector size | Pages | V1 | V2 | V3 |
|---|---|---|---|---|
| 768 | 3,072 px, 27 tiles | 86.4 | 54.8 | 48.7 |
| 512 | 3,072 px, 27 tiles | 86.3 | 54.3 | 48.4 |
| 320 | 3,072 px, 27 tiles | 86.0 | 54.2 | 48.2 |
| 256 | 3,072 px, 27 tiles | 86.0 | 54.3 | 48.0 |
| 128 | 3,072 px, 27 tiles | 85.7 | 54.3 | 46.8 |
| 768 | 2,048 px, 16 tiles | 86.3 | 53.3 | 47.6 |
| 512 | 2,048 px, 16 tiles | 86.3 | 52.9 | 47.5 |
| 320 | 2,048 px, 16 tiles | 86.4 | 52.4 | 47.2 |
| 256 | 2,048 px, 16 tiles | 86.4 | 52.1 | 46.9 |
| 128 | 2,048 px, 16 tiles | 85.6 | 52.6 | 45.9 |
Limits
- A page yields up to 1,792 vectors, so a 768-d page index takes up to 2.6 MiB per page in bf16.
License
MIT. UltraBERT-150M and SigLIP2-B/16-512 are Apache-2.0; their license text is in LICENSE-APACHE.