NeoMME (800M): Single-Tower Multimodal-Native Multilingual Foundation Encoder
Model summary
NeoMME is an efficient Multilingual and Multimodal-native foundational Encoder. Text tokens and raw image patches pass through one shared Transformer encoder. NeoMME does not use a separately pretrained vision encoder or a causal language model.
NeoMME-800M is a pretrained encoder backbone and cannot be used on its own for a downstream task. It returns contextual token representations, so users should fine-tune a task-specific head for retrieval, classification, extraction, or another downstream task. For document retrieval, use NeoMME-800M-Retriever.
Usage
Use NeoMME with transformers:
# accelerate is an optional dependency needed only when using device_map="auto".
pip install -U accelerate transformers
The example below generates hidden states for a text document and a document image in one forward pass. The hidden states are not usable as is for a downstream task. If you are looking for retrieval embeddings, you should use the NeoMME-800M-Retriever model instead.
Generate hidden states
import requests
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor
def encode_document_text(processor, text: str) -> str:
return f"{processor.tokenizer.document_token}{text}"
model_id = "Hcompany/NeoMME-800M"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, device_map="auto")
text = "The cat sat on a mat."
image_url = "https://github.com/tonywu71/colpali-cookbooks/blob/main/examples/data/shift_kazakhstan.jpg?raw=true"
image = Image.open(requests.get(image_url, stream=True).raw)
inputs = processor(
text=[
encode_document_text(processor, text),
encode_document_text(processor, processor.image_token),
],
images=[image],
padding=True,
return_tensors="pt",
).to(model.device)
with torch.inference_mode():
outputs = model(**inputs)
text_hidden_states, image_hidden_states = outputs.last_hidden_state
NeoMME was pretrained with a masked discrete-diffusion objective. Therefore, the model can restore masked text tokens given the surrounding text (and, when present, image patches). The example below fills a single mask as a sanity check of that objective. It is not a generative or conversational model.
Masked language modeling
import torch
from transformers import AutoModelForMaskedLM, AutoProcessor
model_id = "Hcompany/NeoMME-800M"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id, device_map="auto")
# Equivalent: "<doc>The capital of <mask> is London."
text = f"{processor.tokenizer.document_token}The capital of {processor.tokenizer.mask_token} is London."
inputs = processor(text=[text], return_tensors="pt").to(model.device)
with torch.inference_mode():
outputs = model(**inputs)
masked_index = (inputs.input_ids[0] == processor.tokenizer.mask_token_id).nonzero().item()
predicted_token_id = outputs.logits[0, masked_index].argmax(dim=-1)
print(processor.tokenizer.decode(predicted_token_id))
Training
NeoMME-800M was pretrained from scratch on multilingual text and visual-text data, including web text, code, math, document pages, captions, and synthetic OCR data. The model learns to restore masked text tokens. For document images paired with transcripts, image patches remain visible and the pretraining objective has no pixel reconstruction loss.
The NeoMME technical report describes the full pretraining recipe.
Limitations
- NeoMME-800M is a pretrained encoder backbone and requires task-specific fine-tuning.
- The model has not received a comprehensive safety, bias, or privacy evaluation.
License
Model weights are released under the Apache 2.0 license.
Citation
@misc{lac2026neommesingletowermultimodalnativemultilingual,
title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},
author={Aurélien Lac and Tony Wu},
year={2026},
eprint={2609.01657},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2609.01657},
}