NeoMME-Retriever (800M): Single-Tower Multimodal-Native Multilingual Foundation Encoder 🔎

[!IMPORTANT] NeoMME-Retriever (800M) variants:

• Default (transformers) [current]: Returns dense and multi-vector embeddings together with a single forward pass. Recommended for most use cases and inference.
• ST dense: Supports independent dense fine-tuning with Sentence Transformers.
• ST late-interaction: Supports independent multi-vector fine-tuning with Sentence Transformers.

Model summary

NeoMME-800M-Retriever is a model for multimodal document retrieval. Fine-tuned from NeoMME-800M, it encodes text queries and documents (text or page screenshots) using one shared bidirectional Transformer encoder.

A single forward pass returns both multi-vector and dense representations. Multi-vector embeddings use MeanMaxSim scoring, while dense embeddings use cosine similarity.

Performance

All scores use the metric shown at the full trained dimensions. Higher is better. ViDoRe v3, v2, and v1 measure visual document retrieval, while BEIR-15 measures text retrieval.

Usage

Use NeoMME-Retriever with transformers. MeanMaxSim scoring requires sentence-transformers>=6.0.0:

# accelerate is an optional dependency needed only when using device_map="auto".
pip install -U accelerate transformers "sentence-transformers>=6.0.0"

The example below scores a text query against document-page images with late interaction, then with dense cosine similarity. One forward pass returns both embeddings.

from typing import Any, Literal

import requests
import torch
from PIL import Image
from sentence_transformers.util import cos_sim, mean_maxsim

from transformers import BatchFeature, NeoMMEForRetrieval, NeoMMEProcessor


def encode(
    messages: list[list[dict[str, Any]]],
    task: Literal["query", "document"],
) -> BatchFeature:
    return processor.apply_chat_template(
        messages,
        task=task,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        processor_kwargs={"padding": "longest"},
    )


model_name = "Hcompany/NeoMME-800M-Retriever"
processor = NeoMMEProcessor.from_pretrained(model_name)
model = NeoMMEForRetrieval.from_pretrained(model_name, device_map="auto")

# Document images (our corpus)
image_urls = [
    "https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/shift_kazakhstan.jpg?raw=true",
    "https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/energy_electricity_generation.jpg?raw=true",
]
documents = [Image.open(requests.get(url, stream=True).raw) for url in image_urls]

# Queries
queries = [
    "Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?",
    "Which hour of the day had the highest overall electricity generation in 2019?",
]

document_messages = [
    [{"role": "user", "content": [{"type": "image", "image": document}]}] for document in documents
]
query_messages = [[{"role": "user", "content": query}] for query in queries]

inputs_documents = encode(document_messages, "document").to(model.device)
inputs_text = encode(query_messages, "query").to(model.device)

with torch.inference_mode():
    document_outputs = model(**inputs_documents)
    query_outputs = model(**inputs_text)

late_scores = mean_maxsim(
    query_outputs.embeddings,
    document_outputs.embeddings,
    a_mask=inputs_text["attention_mask"],
    b_mask=inputs_documents["attention_mask"],
)
dense_scores = cos_sim(query_outputs.dense_embeddings, document_outputs.dense_embeddings)

# Expected: late_scores[0, 0] > late_scores[0, 1] and late_scores[1, 1] > late_scores[1, 0].
print(late_scores, dense_scores)

The score tensors have shape (num_queries, num_documents) and late_scores[i, j] / dense_scores[i, j] is the score between query i and document j. A larger value indicates a closer match.

Training

NeoMME-800M-Retriever was fine-tuned from NeoMME-800M on text retrieval and document-page images. Training uses a joint late-interaction and Matryoshka dense contrastive objective.

The NeoMME technical report describes the full fine-tuning recipe.

License

Model weights are released under the Apache 2.0 license.

Citation

@misc{lac2026neommesingletowermultimodalnativemultilingual,
      title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},
      author={Aurélien Lac and Tony Wu},
      year={2026},
      eprint={2609.01657},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2609.01657},
}