Nemotron ColEmbed V2

Model Overview

Description

Introducing nvidia/nemotron-colembed-vl-8b-v2, the state-of-the-art late interaction embedding model that currently ranks No. 1 on ViDoRe V3—a comprehensive benchmark for enterprise retrieval use cases—as of January 26, 2026, with a score of 63.54 across 8 public tasks. The model was fine-tuned for query-document retrieval. Users can input queries, which are text, or documents which are page images, to the model. The model outputs ColBERT-style multi-vector numerical representations for input queries and documents.

✨ Key Improvements:

• ⚗️ Advanced Model Merging: Utilizes post-training model merging to combine the strengths of multiple fine-tuned checkpoints. This delivers the accuracy stability of an ensemble without any additional inference latency.
• 🌍 Enhanced Synthetic Data: We significantly enriched our training mixture with diverse multilingual synthetic data, improving semantic alignment across languages and complex document types.

This model is for non-commercial/research use only.
See the Nemotron ColEmbed v2 paper for more details about its architecture, training and results.

License/Terms of Use

The use of this model is governed by the Creative Commons Attribution-NonCommercial 4.0 license and the use of the post-processing scripts are licensed under Apache 2.0. Additional Information: Built with Qwen3-VL which is released under Apache 2.0.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Deployment Geography

Global

Use Case

nemotron-colembed-vl-8b-v2 is a powerful, highly accurate model intended for researchers exploring applications that must understand or retrieve information across both text and image modalities. It is instrumental in multimodal RAG systems, where queries are in text format and documents are images, such as pages, text, charts, tables, or infographics. Potential applications include multimedia search engines, cross-modal retrieval systems, and conversational AI with rich input understanding.

Release Date

01/26/2026 via https://huggingface.co/nvidia/nemotron-colembed-vl-8b-v2

Model Architecture

• Architecture Type: Transformer
• Network Architecture: Qwen3-VL-8B-Instruct based encoder.

The nemotron-colembed-vl-8b-v2 is a transformer-based multimodal embedding model built from Qwen3-VL-8B-Instruct, which adopts a three-module architecture comprising a vision encoder (default to the SigLIP2-SO-400M variant), an MLP-based vision–language merger, and a large language model (LLM) (see technical report for details). It has approximately 8.8B parameters.

Input(s):

Input Type(s): Image, Text

Input Format(s):

• Image: List of images- Red, Green, Blue (RGB)
• Text: List of Strings

Input Parameters:

• Image: Two-Dimensional (2D)
• Text: One-Dimensional (1D)

Other Properties Related to Input:

• The model's maximum context length we evaluated is 10240 tokens.
• Each image tile consumes 256 tokens. We have tested this model extensively with these settings on config.json - max_input_tiles = 8, use_thumbnails = True, so that every image is split into maximum of 8 tiles + 1 thumbnail (whole image at lower resolution). Images must be python PIL format. The model will scale the image into multiple tiles of 512x512.

Outputs

• Output Type: Floats
• Output Format: List of float arrays
• Output Parameters: The list of floats equivalent to [batchsize x seq length x embedding_dim]
• Other Properties Related to Output: For each input token, the model outputs a 4096-dimensional embedding vector of floating-point values.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Installation

The model requires transformers>=4.57.2 and flash attention installed.

pip install "transformers>=4.57.2"
pip install flash-attn==2.6.3 --no-build-isolation

Depending on your environment you might need to upgrade polars and pydantic:

pip install -U datasets polars
pip install -U pydantic

Transformers Usage

We provide an example notebook demonstrating model usage with a simple indexing and retrieval pipeline.

We also provide a simple example code snippet below.

import requests
from PIL import Image
from io import BytesIO
import torch
from transformers import AutoModel
from transformers.image_utils import load_image
# Load Model

model = AutoModel.from_pretrained(
    'nvidia/nemotron-colembed-vl-8b-v2',
    device_map='cuda',
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2"
).eval()

# Queries
queries = [
    'How is AI improving the intelligence and capabilities of robots?',
    'Canary, a multilingual model that transcribes speech in English, Spanish, German, and French with punctuation and capitalization.',
    'Generative AI can generate DNA sequences that can be translated into proteins for bioengineering.'
]

image_urls = [
    "https://developer.download.nvidia.com/images/isaac/nvidia-isaac-lab-1920x1080.jpg",
    "https://developer-blogs.nvidia.com/wp-content/uploads/2024/03/asr-nemo-canary-featured.jpg",
    "https://blogs.nvidia.com/wp-content/uploads/2023/02/genome-sequencing-helix.jpg"
]

# Load all images (load_image handles both local paths and URLs)
images = [load_image(img_path) for img_path in image_urls]

# Encoding
query_embeddings = model.forward_queries(queries, batch_size=8)
image_embeddings = model.forward_images(images, batch_size=8)

scores = model.get_scores(
    query_embeddings,
    image_embeddings
)
# Diagonal should have higher scores
print(scores)

# tensor([[21.1969, 20.9079, 20.5363],
#         [32.2324, 32.8058, 32.2075],
#         [25.7505, 25.8201, 26.2726]], device='cuda:0')

Software Integration:

Runtime Engine(s): Not Applicable Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere - A100 40GB and A100 80GB
• NVIDIA Hopper - H100 80GB

Supported Operating System(s): Linux

Model Version(s)

nemotron-colembed-vl-8b-v2

Training and Evaluation Datasets

Training Dataset

The model was trained on publicly available datasets, including DocMatix-IR, VDR, Vidore-ColPali-Training, VisRAG-Ret-Train-Synthetic-data, VisRAG-Ret-Train-In-domain-data, and Wiki-SS-NQ.

Data Modality: Image

Image Training Data Size

• Less than a Million Images

Data Collection Method by dataset: Hybrid: Automated, Human, Synthetic Labeling Method by dataset: Hybrid: Automated, Human, Synthetic Properties: Training: The vision embedding model was fine-tuned on approximately 500k image samples.

Evaluation Dataset

We evaluate the model on the datasets from ViDoRe V1, V2 and V3 Visual Document Retrieval benchmarks.

ViDoRe is a premier benchmark for Visual Document Retrieval and it is composed of various page-level retrieving tasks spanning multiple domains, languages, and settings. The latest version of the benchmark is Vidore V3, a comprehensive evaluation of retrieval for enterprise use-cases.

We provide a script using MTEB 2 library to evaluate ColEmbed models on ViDoRe benchmarks.

• Data Collection Method by dataset: Hybrid: Automated, Human, Synthetic
• Labeling Method by dataset: Hybrid: Automated, Human, Synthetic
• Properties: More details on ViDoRe V1 and ViDoRe V2 can be found on their leaderboard. Visual Document Retrieval Benchmark,

Evaluation Results

ViDoRE V1&V2 and V3 on MTEB leaderboards

pip install "mteb>=2.7.0, <3.0.0"
# Evaluates with Vidore V1 and V2
CUDA_VISIBLE_DEVICES=0; python3 mteb2_eval.py --model_name nvidia/nemotron-colembed-vl-8b-v2 --batch_size 16 --benchmark "VisualDocumentRetrieval"
# Evaluates with Vidore V3
CUDA_VISIBLE_DEVICES=0; python3 mteb2_eval.py --model_name nvidia/nemotron-colembed-vl-8b-v2 --batch_size 16 --benchmark "ViDoRe(v3)"
# Evaluates with a specific task/dataset of Vidore V3: Vidore3ComputerScienceRetrieval
CUDA_VISIBLE_DEVICES=0; python3 mteb2_eval.py --model_name nvidia/nemotron-colembed-vl-8b-v2 --batch_size 16 --benchmark "ViDoRe(v3)" --task-list Vidore3ComputerScienceRetrieval

In this section, we evaluate the performance of nemotron-colembed-vl-8b-v2 against other models that previously achieved top-five rankings on the leaderboards.

We report results on the ViDoRe benchmark suite. The tables below summarize the image-modality accuracy of nemotron-colembed-vl-8b-v2 on the ViDoRe V1, V2, and V3 benchmarks, alongside other NVIDIA nemotron-colembed models. Note that (M)MTEB leaderboards use Borda ranking. Each task acts like a voter that ranks models based on how well they perform. Models earn more points when they rank higher on a task. The model with the most total points across all tasks gets the top overall rank.

ViDoRe V3 (NDCG@10)

ViDoRe V2 (NDCG@5)

ViDoRe V1 (NDCG@5)

Inference:

Acceleration Engine: Not Applicable Test Hardware: A100 40GB, A100 80GB, H100 80GB

Citation

@misc{moreira2026_nemotron_colembed_v2,
      title={Nemotron ColEmbed V2: Top-Performing Late Interaction embedding models for Visual Document Retrieval}, 
      author={Gabriel de Souza P. Moreira, Ronay Ak, Mengyao Xu, Oliver Holworthy, Benedikt Schifferer, Zhiding Yu, Yauhen Babakhin, Radek Osmulski, Jiarui Cai, Ryan Chesler, Bo Liu, Even Oldridge},
      year={2026},
      eprint={2602.03992},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2602.03992}, 
}

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.