topk-embed-v1-xsmall

topk-embed-v1-xsmall is a 0.8B multimodal late-interaction retriever. It uses text queries to search both text documents and images, such as scanned pages, reports, and slides.

Rather than compressing an input into a single vector, it keeps multiple embeddings for text tokens or image patches. Retrieval uses MaxSim scoring: each query vector is matched to its most similar document vector, and those similarities are summed. Higher scores indicate more relevant documents.

Usage with SentenceTransformers

Install the dependencies in requirements.txt. A CUDA GPU with bfloat16 support (Ampere or newer) is required.

from PIL import Image
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder(
    "topk-io/topk-embed-v1-xsmall",
    trust_remote_code=True,
    device="cuda",
)

query_embeddings = model.encode_query(["What was Q3 revenue?"])

# Encode text and image documents in separate batches.
text_embeddings = model.encode_document([
    "Q3 revenue was $12 million, up 20% year over year.",
    "The company opened a new office in October.",
])
with Image.open("page.png") as page:
    image_embeddings = model.encode_document([page.convert("RGB")])

# MaxSim scores: one row per query, one column per document.
text_scores = model.similarity(query_embeddings, text_embeddings)    # (1, 2)
image_scores = model.similarity(query_embeddings, image_embeddings)  # (1, 1)

Default limits are 1024 tokens per query and 8192 tokens per text document. Embeddings have 1024 dimensions. For smaller Matryoshka (MRL) embeddings, pass config_kwargs={"output_dim": 256} when loading the model.

Evaluation

Results on the eight public ViDoRe v3 test datasets, reported as percentages (higher is better). Columns are Matryoshka (MRL) prefix dimensions; 1024 is the full embedding width. In each per-dataset table, the final row is the unweighted mean across datasets, computed before rounding.

Native queries use the document language: French for Energy, Finance (French), and Physics; English for the remaining datasets. Crosslingual results include all six query languages, including the native language. Image retrieval uses page images; Markdown retrieval uses the corresponding page text.

The per-dataset tables use all document vectors (no token pooling), prefix truncation followed by FP32 L2 normalization, FP16 query/document vector storage, and exhaustive FP32 MaxSim scoring. nDCG@10 uses the graded relevance labels with linear gains. Recall@10 divides retrieved relevant documents by all relevant documents for each query (not capped recall).

Vidore V3 (image, native)

nDCG@10

Recall@10

Vidore V3 (image, crosslingual)

nDCG@10

Recall@10

Vidore V3 (markdown, native)

nDCG@10

Recall@10

Vidore V3 (markdown, crosslingual)

nDCG@10

Recall@10

Pooling performance

Each figure shows image and Markdown retrieval side by side at the full MRL width of 1024 dimensions. Scores are percentages, averaged equally across the eight datasets for each task. Solid blue lines with circles show native queries; dashed amber lines with squares show crosslingual queries. Pooling factors use a base-2 logarithmic axis; 1× is the unpooled baseline. Both panels share the same score scale.

Document vectors are pooled at full width using Ward linkage on cosine distances, then L2-normalized without prefix truncation. Pooling factors 2×, 4×, and 8× reduce each document to approximately one-half, one-quarter, and one-eighth as many vectors. Queries are not pooled. Vector storage remains FP16; scoring and the pre-export evaluation settings are the same as above.

nDCG@10

Recall@10