Omni-Embed-Mini-0.9B

935M parameters. Text, speech, general audio, image, video, and visually-rich documents in a single shared cosine space, from a backbone that is never updated.

Project page | Code | Sibling model: Omni-Embed-Mini-2.3B

What this is

Omni-Embed-Mini recasts cross-modal alignment as self-distillation through one shared frozen causal backbone. Every media sample is paired with a dense cascaded caption; the teacher target is the EOS-pooled embedding of that caption produced by the identical frozen backbone that processes the student input. Teacher and student therefore inhabit byte-identical geometry, and because no text-side parameter is ever updated, adding five modalities cannot degrade the text representation.

Only the projectors and small phased LoRA adapters on the modality encoders are trained. A Matryoshka SigLIP contrastive objective and an online hybrid hard-negative miner supply the contrastive signal.

At 935M parameters this is 2.7x to 9.5x smaller than competing open omni embedders.

Results

Six modalities, evaluated with the pipeline in the code repository.

The text score is the headline result: the 0.9B preserves its frozen backbone's MTEB-v2 score while extending it to five further modalities.

Quick start

pip install "transformers>=5.0" torch pillow numpy soundfile huggingface_hub

Qwen/Qwen3.5-0.8B (about 1.7 GB) is fetched on first load to build the visual tower.

import torch
from transformers import AutoModel, AutoProcessor

REPO = "MBZUAI/Omni-Embed-Mini-0.9B"

model = AutoModel.from_pretrained(
    REPO, trust_remote_code=True, dtype=torch.bfloat16,
).cuda().eval()
processor = AutoProcessor.from_pretrained(REPO, trust_remote_code=True)

# OpenAI-style multimodal messages: text, audio, image, video, doc, or any composition.
messages = [{"role": "user", "content": [
    {"type": "audio", "audio": "path/to/clip.wav"},
    {"type": "text",  "text":  "rain on a tin roof at night"},
]}]

inputs = processor.apply_chat_template(
    messages, role="passage", tokenize=True, return_tensors="pt",
).to("cuda")
with torch.no_grad():
    doc = model(**inputs).pooler_output          # (1, 1024), already L2-normalised

# Query side. role="query" applies the retrieval instruction template.
# text_recipe="chat" puts a text query in the same subspace as media documents;
# omit it only for pure text-to-text retrieval, which uses the native recipe.
query = processor.apply_chat_template(
    [{"role": "user", "content": "rain on a tin roof at night"}],
    role="query", tokenize=True, return_tensors="pt", text_recipe="chat",
).to("cuda")
with torch.no_grad():
    q = model(**query).pooler_output

print("cosine:", float(q @ doc.T))               # normalised, so dot == cosine

Matryoshka (truncatable) embeddings

model(**inputs, truncate_dim=N).pooler_output returns a prefix-truncated, renormalised vector. Slicing pooler_output yourself is not equivalent unless you renormalise after. Supported N: 128, 256, 512, 1024. Use a smaller N to cut index size at a modest recall cost.

Architecture

Trained parameters: audio and vision projectors plus phase-2 LoRA adapters on the modality encoders (48 Whisper, 24 Dasheng, and 24 vision adapted tensors, merged into the released weights). The backbone is frozen at every stage and ships unmodified apart from a vocabulary resize that adds the media placeholder tokens.

Files

config.json                 modeling_omni_embed.py      processing_omni_embed.py
configuration_omni_embed.py chat_template.jinja         processor_config.json
tokenizer.json              tokenizer_config.json
backbone/                   # frozen backbone (vocab-resized), safetensors
whisper_encoder.pt          dasheng_encoder.pt          vision_encoder.pt
projector_weights.pt        # projectors; LoRA already merged into the encoders

The .pt files are PyTorch pickles, and the custom modeling code requires trust_remote_code=True. Load only from a source you trust.

Citation

@inproceedings{kurpath-etal-2026-omni-embed-mini,
    title = "{O}mni-{E}mbed-{M}ini: Binding Modalities Without Forgetting via Dense Distillation",
    author = "Kurpath, Mohammed Irfan and Kaithakkodan, Jaseel Muhammad and Mullappilly, Sahal Shaji and Laptev, Ivan and Cholakkal, Hisham",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2026",
    year = "2026",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2610.02148"
}