llama-nemoretriever-colembed-3b-v1

Description

The nvidia/llama-nemoretriever-colembed-3b-v1 is a late interaction embedding model fine-tuned for query-document retrieval. Users can input queries, which are text, or documents which are page images, to the model. The model outputs ColBERT-style multi-vector numerical representations for input queries and documents. It achieved 1st place on ViDoRe V1 (nDCG@5), ViDoRe V2 (nDCG@5) and MTEB VisualDocumentRetrieval (Rank Borda) (as of 27th June, 2025).

For more technical details, please refer to our technical report: Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model

This model is for non-commercial/research use only.

License/Terms of Use

Governing Terms for llama-nemoretriever-colembed-3b-v1 model: NVIDIA Non-Commercial License
Additional Information: Apache License 2.0 for siglip2-giant-opt-patch16-384; and LLAMA 3.2 Community License Agreement for Llama-3.2-1B. Built with Meta Llama 3. Improved using Qwen.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Team

• Mengyao Xu
• Gabriel Moreira
• Radek Osmulski
• Ronay Ak
• Yauhen Babakhin
• Even Oldridge
• Benedikt Schifferer

Citation

@misc{xu2025llamanemoretrievercolembedtopperforming,
      title={Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model}, 
      author={Mengyao Xu and Gabriel Moreira and Ronay Ak and Radek Osmulski and Yauhen Babakhin and Zhiding Yu and Benedikt Schifferer and Even Oldridge},
      year={2025},
      eprint={2507.05513},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2507.05513}, 
}

@misc{li2025eagle2buildingposttraining,
      title={Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models}, 
      author={Zhiqi Li and Guo Chen and Shilong Liu and Shihao Wang and Vibashan VS and Yishen Ji and Shiyi Lan and Hao Zhang and Yilin Zhao and Subhashree Radhakrishnan and Nadine Chang and Karan Sapra and Amala Sanjay Deshmukh and Tuomas Rintamaki and Matthieu Le and Ilia Karmanov and Lukas Voegtle and Philipp Fischer and De-An Huang and Timo Roman and Tong Lu and Jose M. Alvarez and Bryan Catanzaro and Jan Kautz and Andrew Tao and Guilin Liu and Zhiding Yu},
      year={2025},
      eprint={2501.14818},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2501.14818}, 
}

@misc{moreira2025nvretrieverimprovingtextembedding,
      title={NV-Retriever: Improving text embedding models with effective hard-negative mining}, 
      author={Gabriel de Souza P. Moreira and Radek Osmulski and Mengyao Xu and Ronay Ak and Benedikt Schifferer and Even Oldridge},
      year={2025},
      eprint={2407.15831},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2407.15831}, 
}

NVIDIA’s Retrieval Models

Deployment Geography

Global

Use Case

llama-nemoretriever-colembed is intended for researchers exploring applications that must understand or retrieve information across both text and image modalities. It is instrumental in multimodal RAG systems, where queries are in text format and documents are images, such as pages, text, charts, tables or infographics. Potential applications include multimedia search engines, cross-modal retrieval systems, and conversational AI with rich input understanding.

Release Date

Huggingface on 06/27/2025 via https://huggingface.co/nvidia/llama-nemoretriever-colembed-3b-v1

Model Architecture

• Architecture Type: Transformer
• Network Architecture: google/siglip2-giant-opt-patch16-384 + meta-llama/Llama-3.2-3B

The llama-nemoretriever-colembed-3b-v1 is a transformer-based multimodal embedding model built on top of a VLM based on google/siglip2-giant-opt-patch16-384 and meta-llama/Llama-3.2-3B.

Input

Output

• Output Type: Floats
• Output Format: List of float arrays
• Output Parameters: The list of floats equivalent to [batchsize x seq length x embedding_dim]
• Other Properties Related to Output: Model outputs embedding vectors of dimension for each input token.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Usage

The model requires transformers version 4.49.0 and flash attention

pip install transformers==4.49.0
pip install flash-attn==2.6.3 --no-build-isolation

import requests
from PIL import Image
from io import BytesIO
import torch
from transformers import AutoModel

# Load Model
model = AutoModel.from_pretrained(
    'nvidia/llama-nemoretriever-colembed-3b-v1',
    device_map='cuda',
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2"
).eval()

# Queries
queries = [
	'How much percentage of Germanys population died in the 2nd World War?',
	'How many million tons CO2 were captured from Gas processing in 2018?',
	'What is the average CO2 emission of someone in Japan?'
]

# Documents
image_urls = [
	'https://upload.wikimedia.org/wikipedia/commons/3/35/Human_losses_of_world_war_two_by_country.png',
	'https://upload.wikimedia.org/wikipedia/commons/thumb/7/76/20210413_Carbon_capture_and_storage_-_CCS_-_proposed_vs_implemented.svg/2560px-20210413_Carbon_capture_and_storage_-_CCS_-_proposed_vs_implemented.svg.png',
	'https://upload.wikimedia.org/wikipedia/commons/thumb/f/f3/20210626_Variwide_chart_of_greenhouse_gas_emissions_per_capita_by_country.svg/2880px-20210626_Variwide_chart_of_greenhouse_gas_emissions_per_capita_by_country.svg.png'
]

# Load into PIL
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
}

images = [Image.open(BytesIO(requests.get(image_url, headers=headers).content)) for image_url in image_urls]

# Encoding
query_embeddings = model.forward_queries(queries, batch_size=8)
image_embeddings = model.forward_images(images, batch_size=8)


scores = model.get_scores(
    query_embeddings,
    image_embeddings
)
# Diagonal should have high scores
print(scores)
# tensor([[13.9970, 11.4219, 12.1225],
#         [11.4157, 14.6388, 12.0341],
#         [ 9.9023,  9.8857, 11.3387]], device='cuda:0')

Running evaluation on benchmarks

Vidore V1&V2 and V3 on MTEB leaderboards

ViDoRe is a premier benchmark for Visual Document Retrieval. The latest version of the benchmark is Vidore V3, a comprehensive evaluation of retrieval for enterprise use-cases.
We provide a script script using MTEB 2 library to evaluate ColEmbed models on ViDoRe benchmarks.

pip install "mteb>=2.3.10,<3.0.0"
# Evaluates with Vidore V1 and V2
CUDA_VISIBLE_DEVICES=0; python3 mteb2_eval.py --model_name nvidia/llama-nemoretriever-colembed-3b-v1 --batch_size 16 --benchmark "VisualDocumentRetrieval"
# Evaluates with Vidore V3
CUDA_VISIBLE_DEVICES=0; python3 mteb2_eval.py --model_name nvidia/llama-nemoretriever-colembed-3b-v1 --batch_size 16 --benchmark "ViDoRe(v3)"
# Evaluates with a specific task/dataset of Vidore V3: Vidore3ComputerScienceRetrieval
CUDA_VISIBLE_DEVICES=0; python3 mteb2_eval.py --model_name nvidia/llama-nemoretriever-colembed-3b-v1 --batch_size 16 --benchmark "ViDoRe(v3)" --task-list Vidore3ComputerScienceRetrieval

We also provide a script to evaluate ViDoRe V1 and ViDoRe V2 based on their deprecated GitHub repository.

pip install git+https://github.com/illuin-tech/vidore-benchmark@e0eb9032e7e00adc8aa6f9cb35d5a9371f67485a
# Downgrade transformers as vidore will install latest transformers
pip install --upgrade transformers==4.49.0
CUDA_VISIBLE_DEVICES=0; python3 vidore_eval.py --model_name_or_path nvidia/llama-nemoretriever-colembed-3b-v1 --savedir_datasets ./results/ --model_revision 50c36f4d5271c6851aa08bd26d69f6e7ca8b870c

Software Integration:

Runtime Engine(s): TensorRT, Triton Supported Hardware Microarchitecture Compatibility: A100 40GB, A100 80GB, H100 80GB Supported Operating System(s): Linux

Model Version(s)

llama-NemoRetriever-colembed-3b-v1

Training and Evaluation Datasets

• The total size (in number of data points) 12.74M qa pairs for training
• Total number of datasets: 23 datasets used for training and 17 datasets used for evaluation.

Training Dataset

The model was trained on publicly available datasets, including HotpotQA, MIRACL, Natural Questions (NQ), Stack Exchange, SQuAD, Tiger Math/Stack, DocMatix-IR, VDR, Vidore-ColPali-Training, VisRAG-Ret-Train-Synthetic-data, VisRAG-Ret-Train-In-domain-data, and Wiki-SS-NQ.

• Data Collection Method by dataset: Hybrid: Automated, Human, Synthetic
• Labeling Method by dataset: Hybrid: Automated, Human, Synthetic
• Properties: Training: 1st Stage: 12M qa pairs, 2nd Stage: 500k qa pairs, 3rd Stage: 240k qa pairs

Evaluation Dataset

We evaluate the model on multiple benchmarks for Visual Document Retrieval, ViDoRe V1, ViDoRe V2 and MTEB Visual Document Retrieval.

• Data Collection Method by dataset: Hybrid: Automated, Human, Synthetic
• Labeling Method by dataset: Hybrid: Automated, Human, Synthetic
• Properties: More details on ViDoRe V1 and ViDoRe V2 can be found on their leaderboard. Visual Document Retrieval Benchmark, ViDoRe, is composed of various page-level retrieving tasks spanning multiple domains, languages, and settings.

Note: All scores are Avg. NDCG@5. ViDoRe V1 and V2 was updated on June 27th 2025 to use the calculated scores from MTEB, which can result in slightly different scores. The ViDoRe V2 (06/27/2025) uses only 4 of the original 7 datasets.

Inference:

Acceleration Engine: Not Applicable Test Hardware: A100 40GB, A100 80GB, H100 80GB

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns here.