jina-embeddings-v5-omni-nano-clustering-GGUF

jina-embeddings-v5-omni-nano-clustering-GGUF: Clustering-Targeted Omni Embedding (Nano) — GGUF

ArXiv | Blog

Average score vs. parameter count across image (MIEB-Lite), video (MMEB-V), and audio (MAEB) benchmarks — jina-v5-omni-nano and jina-v5-omni-small define the open-weight frontier (Table 1 in the ArXiv report).

Model Overview

GGUF + multimodal-projector build of jinaai/jina-embeddings-v5-omni-nano-clustering for llama.cpp. Accepts text, images, video, and audio and produces 768-dim embeddings in the same vector space as the torch reference and as jinaai/jina-embeddings-v5-text-nano-clustering at the same task — index with text and query with any modality, no reindexing. For higher performance at a larger size, see jinaai/jina-embeddings-v5-omni-small-clustering-GGUF.

This is the clustering-targeted variant of the jina-embeddings-v5-omni-nano GGUF family. The umbrella with all GGUF variants and cross-repo benchmarks is jina-ai/jina-embeddings-v5-omni-gguf.

Cross-repo docs: benchmarks (NDCG@5 on NanoBEIR, tokens/sec, peak VRAM, file size), the full per-variant numerical-parity tables, and runtime caveats live in the v5-omni-gguf umbrella.

Via Elastic Inference Service

The fastest way to use v5-omni in production. Elastic Inference Service (EIS) provides managed embedding inference with built-in scaling, so you can generate embeddings directly within your Elastic deployment.

# Retrieve the configuration of the preconfigured omni-nano inference endpoint
GET /_inference/embedding/.jina-embeddings-v5-omni-nano

# Generate an embedding for a single piece of text using the predefined endpoint
POST _inference/embedding/.jina-embeddings-v5-omni-nano
{
  "input": [
    "This is a test"
  ]
}

# Fuse a text description and an image into a single embedding via a multimodal content block
POST _inference/embedding/.jina-embeddings-v5-omni-nano
{
  "input": [
    {
      "content": [
        { "type": "text",  "value": "A small blue square" },
        { "type": "image", "format": "base64", "value": "<BASE64_IMAGE_DATA>" }
      ]
    }
  ]
}

# Create a custom endpoint that truncates omni-nano embeddings to 32 dimensions
PUT _inference/embedding/jina-omni-nano-32d
{
  "service": "elastic",
  "service_settings": {
    "model_id": "jina-embeddings-v5-omni-nano",
    "dimensions": 32
  }
}

See the Elastic Inference Service documentation for setup details.

Files in this repo

llama.cpp loads --mmproj to enable image / video / audio inputs on top of the text GGUF. The two mmprojs are independent — load whichever modality you need, or pass --mmproj twice to serve both from one process (see "Selective modality loading" below).

Install llama.cpp (with multimodal patches)

This model relies on the Jina v5 omni patches (audio chunked attention, qwen3vl video temporal-pair, encoder combined-decode, etc.) — they are not yet upstream. Build from the feat-v5-omni fork:

git clone https://github.com/jina-ai/llama.cpp.git
cd llama.cpp
git checkout feat-v5-omni
cmake -B build && cmake --build build --config Release -j

For CUDA: pass -DGGML_CUDA=ON to the configure step.

H100 / Hopper note. On Hopper GPUs (H100, H200), set GGML_CUDA_DISABLE_GRAPHS=1 before launching llama-server. Without it, the CUDA-graph capture/replay path crashes with cudaMemcpyAsync … illegal instruction during embedding extraction. CPU, Metal, Vulkan, and pre-Hopper CUDA (e.g. L4, A100) are unaffected.

Quickstart — text via llama-embedding

./build/bin/llama-embedding \
    -hf jinaai/jina-embeddings-v5-omni-nano-clustering-GGUF:Q4_K_M \
    --pooling last --embd-normalize 2 \
    -p "A cute cat sitting on a mat."

The -hf shortcut downloads + caches the requested quant from this repo on first use. Q5_K_M is the recommended default for nano; Q8_0 for highest fidelity; IQ2_*/IQ1_* for very tight memory budgets.

No prefix convention. Clustering text is embedded verbatim — no Query: / Document: prefixes are needed (unlike the retrieval variant). Both sides of any text pair go in unprefixed.

No custom pooling or padding code needed — --pooling last and --embedding are the only flags required; min_pixels / max_pixels / temporal_patch_size are baked into the GGUF metadata and the mmproj.

Quickstart — text + image via llama-server

Start the server with the vision mmproj:

./build/bin/llama-server \
    -m jina-embeddings-v5-omni-nano-clustering-Q5_K_M.gguf \
    --mmproj jina-embeddings-v5-omni-nano-clustering-vision-mmproj-F16.gguf \
    --embedding --pooling last \
    --host 127.0.0.1 --port 8080

POST to /embeddings with the v5-omni image prompt template:

import base64, requests

with open("photo.jpg", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

# text query
q = requests.post("http://127.0.0.1:8080/embeddings", json={
    "content": [{"prompt_string": "A cute cat sitting on a mat."}]
}).json()[0]["embedding"]

# image embedding (one base64-encoded image per <__media__> marker)
i = requests.post("http://127.0.0.1:8080/embeddings", json={
    "content": [{
        "prompt_string": "<__media__>",
        "multimodal_data": [img_b64],
    }]
}).json()[0]["embedding"]

The <__media__> placeholder is replaced server-side with the right sequence of image tokens.

Quickstart — text + video via llama-server

Same vision mmproj, but use videopair_data to pass frame pairs (temporal_patch_size=2, matching torch's 3D conv with kt=2):

import base64, imageio.v3 as iio, requests

frames = list(iio.imiter("clip.mp4"))     # decode video → list of HxWx3 frames
def b64(arr):
    import io, numpy as np
    from PIL import Image
    buf = io.BytesIO(); Image.fromarray(np.asarray(arr)).convert("RGB").save(buf, "PNG")
    return base64.b64encode(buf.getvalue()).decode()

# group consecutive frames into pairs
pairs = [(b64(frames[i]), b64(frames[i+1])) for i in range(0, len(frames) - 1, 2)]
prompt = "<__media__>" * len(pairs)       # one marker per logical (paired) frame

v = requests.post("http://127.0.0.1:8080/embeddings", json={
    "content": [{"prompt_string": prompt, "videopair_data": pairs}]
}).json()[0]["embedding"]

Quickstart — text + audio via llama-server

Start a server with the audio mmproj (or run a second instance on a different port if you already have a vision server up):

./build/bin/llama-server \
    -m jina-embeddings-v5-omni-nano-clustering-Q5_K_M.gguf \
    --mmproj jina-embeddings-v5-omni-nano-clustering-audio-mmproj-F16.gguf \
    --embedding --pooling last \
    -b 4096 -ub 4096 \
    --host 127.0.0.1 --port 8081

The -b 4096 -ub 4096 flags bump the physical batch size since audio prompts can expand to ~750 tokens for a 30s clip.

import base64, requests
with open("speech.wav", "rb") as f:
    a_b64 = base64.b64encode(f.read()).decode()

a = requests.post("http://127.0.0.1:8081/embeddings", json={
    "content": [{
        "prompt_string": "<__media__>",
        "multimodal_data": [a_b64],
    }]
}).json()[0]["embedding"]

WAV / MP3 / FLAC are accepted; audio is resampled internally to 16kHz mono. For an 11s clip the runtime emits 275 audio tokens; for a 30s clip, 750.

Selective modality loading (text / vision / audio / omni)

Mirrors the HF modality= argument. Pass at most one vision mmproj and at most one audio mmproj — the fork accepts --mmproj more than once:

Combined invocation:

./build/bin/llama-server \
    -m         jina-embeddings-v5-omni-nano-clustering-Q5_K_M.gguf \
    --mmproj   jina-embeddings-v5-omni-nano-clustering-vision-mmproj-F16.gguf \
    --mmproj   jina-embeddings-v5-omni-nano-clustering-audio-mmproj-F16.gguf \
    --embedding --pooling last \
    --host 127.0.0.1 --port 8080 \
    -b 8192 -ub 8192

Vision and audio embeddings produced this way are bit-identical to the single-mmproj invocations — the encoder graph is the same regardless of whether the other modality's projector is also loaded.

Matryoshka (truncating embeddings)

Any prefix of the output vector is itself a valid embedding once L2-renormalized. Supported prefix dims: {32, 64, 128, 256, 512, 768}. Verified end-to-end through the GGUF encode pipeline: prefix dims produce vectors with cos-vs-torch matching the full vector to within quantization noise (max prefix-vs-full drift: +0.0000 at F16, +0.0068 at Q4_K_M for this nano model).

import numpy as np
full = np.asarray(v)
truncated = full[:256]
truncated /= np.linalg.norm(truncated)

Text quantization levels

F16 + 13 int-quant levels, imatrix-calibrated against a multilingual text corpus (calibration_data_v5_rc.txt). Numbers below are min cos vs torch fp32 across the 7-input reference set in ref_nano_clustering.json, bucketed by token length (very_short = 2-4 tokens, short = 5-15, medium = 16-30):

Recommendation: Q5_K_M is the production CPU default for nano. Higher levels (Q6_K / Q8_0) are conservative choices when very-short or multilingual inputs (titles, single-word queries) dominate. IQ-quants and Q2_K and below break down on tiny inputs — use only for memory-constrained testing.

The vision and audio mmprojs ship in F16 only — quantization beyond F16 on the projector tensors causes large parity loss and is not worth the disk savings.

Batching

llama-server's /embeddings endpoint accepts a list of inputs in the content array — one forward pass per element, returned as separate embeddings:

import requests
batch = requests.post("http://127.0.0.1:8080/embeddings", json={
    "content": [
        {"prompt_string": "A cute cat sitting on a mat."},
        {"prompt_string": "A red sports car parked under a tree."},
    ]
}).json()
# batch[0]["embedding"], batch[1]["embedding"]

Multimodal inputs are forwarded per-sample (one pass per image / video / audio). Long text-only batches benefit most from -b 8192 -ub 8192. For high-throughput multimodal serving, prefer the vLLM path on the torch base model.

Multimodal parity vs torch (cos ≥ 0.99 numerical bar)

Verified on the same fork build that produces this repo:

Cross-variant comparison: see the v5-omni-gguf umbrella.

Compatibility

Same vector space as:

• jinaai/jina-embeddings-v5-text-nano-clustering — text-only
• jinaai/jina-embeddings-v5-omni-nano-clustering — multimodal (transformers / ST / vLLM)

Notes

• Last-token pooling is used throughout (matches the torch reference).
• For best parity on nano, prefer Q5_K_M or higher for the text quantization (Q4_K_M drops very-short inputs below 0.9 cos); the mmprojs ship F16 only.
• The <image> token is shared between image and video inputs; video uses image_grid_thw=[T,H,W] with T=2 (the Qwen3-VL ViT's Conv3d patch_embed handles the temporal dimension). The GGUF videopair_data API is identical for image and video paths.

License

CC BY-NC 4.0. For commercial use, contact us.