🏆 EVIE-8B: The Most Accurate Visual Document Retriever

Evidence-Vector-Informed Embedding (EVIE-8B Flagship Teacher)

📢 Release Announcement: All model weights, inference pipelines, and evaluation suites have been fully open-sourced. The technical report is available on arXiv: EVIE: Evidence-Vector-Informed Embeddings for Visual Document Retrieval. Training code and distillation recipes are available in the Tencent/EVIE repository.

🌟 Highlights

• Highest Retrieval Accuracy: 66.75 nDCG@10 on ViDoRe V3, delivering industry-leading visual document retrieval accuracy.
• High-Capacity 4096D Representations: Full per-token multi-vector embeddings preserving fine-grained layout, typography, charts, and table structures.
• Teacher Foundation: Provides capacity-aware relation and margin distillation targets for the lightweight EVIE-4.5B Prefix-MRL model.
• Multi-Benchmark 138-Task Coverage: Thoroughly validated across 138 tasks (ViDoRe V1, V2, V3, and JinaVDR) across 4 standard metric families (nDCG, Recall, MAP, MRR @1/5/10).

🧠 Architecture & Technical Highlights

 Query Text  ────────► ColQwen3.5 (BiDir Attention) ────► 4096D Per-Token Embeddings
                                                                    │
                                                           MaxSim Matching
                                                                    │
 Doc Image   ────────► ColQwen3.5 (Vision Encoder)  ────► Full Multi-Vector Patch Embeddings

• Late-Interaction Token Embeddings: Rather than compressing visual documents into an information-lossy single vector, EVIE-8B preserves spatial nuances, typography, and tabular relations with per-token late-interaction representations. Relevance is scored using MaxSim:

$$ S(Q, D) = \sum_{i=1}^{|Q|} \max_{j=1}^{|D|} (q_i \cdot d_j) $$

• Full Bidirectional Attention: Employs bidirectional self-attention across multimodal vision-text sequences to facilitate cross-modal interaction between visual patch tokens and prompt contexts.
• Foundational Distillation Teacher: Serves as the high-capacity anchor model for training the compact EVIE-4.5B student via topological relation transfer and hard-negative margin supervision (ARD).

🏆 ViDoRe V3 Leaderboard

Official ranking on the ViDoRe leaderboard (ViDoRe V3, Mean Task).

📊 Comprehensive ViDoRe Leaderboard Comparison

Performance comparison across modern multi-vector late-interaction visual document retrievers on ViDoRe:

🔍 ViDoRe V3 Per-Domain Breakdown (nDCG@10)

📋 Complete 138-Task Evaluation Matrix

Protocol paired-all-pages-dedup+process_queries+ndcg2r-20260827 ($\text{MVT} = 1024$, bidirectional attention):

• ViDoRe V1: 10 tasks, 3,943 queries, 6,729 docs
• ViDoRe V2: 4 tasks, 1,152 queries, 4,544 docs
• ViDoRe V3: 48 tasks, 14,514 queries, 115,512 docs
• JinaVDR: 76 tasks, 43,375 queries, 55,398 docs

⚡ Quick Start

ColPali Engine

Installation

git clone https://github.com/Tencent/EVIE.git
cd EVIE
pip install -r requirements.txt
export PYTHONPATH="$(pwd)/colpali${PYTHONPATH:+:$PYTHONPATH}"

Python Inference

import torch
from PIL import Image
from colpali_engine.models import ColQwen3_5, ColQwen3_5Processor

model_id = "tencent/EVIE-8B"

# 1. Load model with FlashAttention and bidirectional attention
model = ColQwen3_5.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    attn_implementation="flash_attention_2",
).eval()
model.enable_bidirectional_attention()

# 2. Process query and document image
processor = ColQwen3_5Processor.from_pretrained(model_id)
images = [Image.open("examples/demo/pages/q3_revenue.png").convert("RGB")]
queries = ["What is the total quarterly revenue?"]

image_batch = processor.process_images(images).to(model.device)
query_batch = processor.process_queries(queries).to(model.device)

# 3. Generate 4096D multi-vectors and late-interaction score
with torch.inference_mode():
    image_embeddings = model(**image_batch)
    model.rope_deltas = None  # Reset RoPE deltas before text query forward
    query_embeddings = model(**query_batch)

scores = processor.score(query_embeddings, image_embeddings)
print("Late-interaction MaxSim Relevance Score:", scores)

Sentence Transformers

Load EVIE with Sentence Transformers to encode queries and document images and compute MaxSim scores. Bidirectional attention is configured automatically.

pip install -U "sentence-transformers[image]>=6.0.0"

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("tencent/EVIE-8B")

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
documents = [
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc3.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc4.jpg",
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents, batch_size=1)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# torch.Size([23, 4096]) torch.Size([3161, 4096])

scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[14.3594,  6.1758,  4.5137,  3.2979],
#         [ 1.8079, 10.9180,  1.9668,  1.9541]])

Documents can be URLs, local image paths, or PIL.Image objects. The output contains 4096-dimensional token embeddings. Scores can vary slightly with dtype and attention backend.

The default page budget allows up to 16,384 visual tokens. Encoding images one at a time reduces peak memory use. To use the 1,024-token budget from the evaluation protocol above, set the image processor's pixel budget:

model = MultiVectorEncoder(
    "tencent/EVIE-8B",
    processor_kwargs={"size": {"longest_edge": 1024 * 32 * 32, "shortest_edge": 65536}},
)

📂 Repository Layout

Evie-8B/
├── model.safetensors         # EVIE-8B weights (safetensors)
├── config.json               # Model configuration (4096D)
├── 1_Dense/                  # Sentence Transformers 4096D projection
├── modules.json              # MultiVectorEncoder module graph
├── infer.py                  # Standalone inference & scoring CLI
├── colpali/                  # ColQwen3.5 inference engine
├── code/shared/              # 138-task eval harness
├── examples/demo/            # 8-page retrieval demo (run.sh)
└── env.sh.example            # Environment variables template

🔬 Released Checkpoint

EVIE-8B was trained on 775,635 document-query pairs across multilingual domains. Hard negatives were mined and classified into answerable (promoted to positives), ambiguous (masked), and strict negatives (negative cross-entropy loss).

The released a40 checkpoint represents a weight-space mix ($\alpha = 0.40$) of two independently trained arms, achieving the peak 66.75 nDCG@10 on ViDoRe V3. Full training reproduction scripts are provided in the companion repository Tencent/EVIE.

📚 Citation

@misc{wang2026evie,
  title         = {EVIE: Evidence-Vector-Informed Embeddings for Visual Document Retrieval},
  author        = {Zifei Wang and Wei Wen and Qiang Ji and Qian-Wen Zhang and Ruizhi Qiao and Xing Sun},
  year          = {2026},
  eprint        = {2610.11553},
  archivePrefix = {arXiv},
  primaryClass  = {cs.IR},
  url           = {https://arxiv.org/abs/2610.11553}
}

📄 License

This model and codebase are released under the Apache-2.0 License.

👥 Authors