Vela Omni Mini

Text, images, speech and environmental sounds in one embedding space. Vela Omni Mini supports multimodal search, routing and clustering with normalized vectors that can be compared directly.

Try Vela Studio · Vela collection · Detailed evaluation

Model overview

Evaluation

Scores are 0–100; higher is better. The comparison uses the same held-out evaluation examples, complete retrieval pools and 128-token text cap for both models. Bold marks improvement over the original large model. These known test pools are reused across releases.

The common protocol uses labeled TRAIN prototypes for text classification and all matching positives for retrieval; it is separate from official MTEB classification. All 14 metrics, Macro-F1, exact counts and uncertainty.

Standard benchmarks and rankings

The primary metric, Mean(TaskType), weights each task type equally. Mean(Task) weights individual tasks equally and is supplementary. Rankings use complete benchmark results from the September 17, 2026 registry snapshots plus both current Vela models; “≤ size” counts models with no more total parameters than this model.

Mini ranks 10/134 on instructed English and 5/50 on audio by Mean(TaskType) among models with no more than its 1.361B total parameters. Its supplementary audio Mean(Task) rank is 3/50. English uses fixed official MTEB task instructions; audio and the scorecard above use the default shared mode. A new comparison with the same text weights and evaluation settings raises Mean(TaskType) from 58.79 to 64.68 (+5.89 pp): 38 tasks improve, three decline, and all seven task types improve. This uses a fresh matched raw control; the historical default result is retained separately. Matched comparison and regressions. Snapshot comparisons include single-modality specialists and reported protocols; the original large has not been evaluated on these complete panels. Full rankings and both aggregate metrics · All task scores and methods.

Audio Mean(TaskType) rises from 52.94 to 54.87 without adding parameters. SpeechCommands zero-shot falls from 40.35 to 38.12, and several speech retrieval and audio classification metrics decline. Every change.

Quality and model size

Each plot combines the full benchmark score, global and size-constrained ranks, and the gap to the best model at no greater total size. Every complete model with a known size is plotted, including models below the observed Pareto frontier. Highlighting Vela does not imply frontier membership. Click either figure for full resolution.

Selected task strengths

These task-level comparisons highlight specific strengths; they do not establish overall benchmark SOTA.

Comparison data and methodology · Figure gallery

Usage

Use PyTorch and a matching torchaudio build, Transformers 4.57.6, Hugging Face Hub, safetensors, NumPy and Pillow. CPU loading was verified with PyTorch/torchaudio 2.8.0:

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("llm-semantic-router/Vela-1.0-Omni-Mini")
sys.path.insert(0, path)
from vela_omni import VelaOmni

model = VelaOmni.from_pretrained(path, device="cpu")
vectors = model.encode_text(["A bicycle beside a building.", "Someone is reading aloud."])
print(vectors.shape)  # (2, 768)

The default mode above keeps text in the shared image/audio space. For text retrieval, opt into an instruction and keep documents unprefixed:

queries = model.encode_text(
    ["How do I reset my password?"],
    instruction="Given a search query, retrieve passages that answer the query.",
    role="query",
)
documents = model.encode_text(["Open Settings and choose Reset password."], role="document")
scores = queries @ documents.T

task="retrieval", role="query" selects a generic preset; custom instruction= and task= are mutually exclusive. The benchmark uses the fixed official task instructions, not a preset sweep. Instructions are intended for text tasks; use the default mode for cross-modal comparisons.

The 32,768-token limit includes the instruction prefix and special tokens. Longer formatted inputs raise ValueError; pass truncate=True only when whole-input truncation is intended. Modes, roles and context rules.

Pass a list of Pillow images to model.encode_image(images). Pass original-rate mono NumPy waveforms and their actual rate, for example model.encode_audio(waveforms, sampling_rate=48000) for 48 kHz recordings. Each waveform must be at most 30 seconds; the model independently derives its 16 kHz speech and 48 kHz sound inputs. Existing 16 kHz recordings are also accepted. Compare normalized vectors with their dot product. To route media, embed each destination’s name and description, then select the closest vector. To discover routing categories, cluster media embeddings and inspect each group. Similarity scores are rankings, not calibrated probabilities.

Training and license

Cross-modal alignment uses COCO image–caption pairs from the CC-BY 2.0 image subset and LibriSpeech speech–transcript pairs (CC-BY 4.0). COCO annotations, Banking77, and MASSIVE are provided under CC-BY 4.0.

The audio weights retain the existing backbones and the residual projection trained with TRAIN-only speech and event geometry constraints. This release adds an optional text API without changing any model weights.

Residual audio alignment additionally uses 3,299 FSD50K TRAIN recordings (CC0 or CC-BY 3.0); per-recording attribution is included. The added CLAP unfused checkpoint is Apache 2.0; its upstream code license is retained separately.

The text component uses Qwen3-Embedding-0.6B, licensed under Apache 2.0. The repository includes the native model code, component configurations, and tokenizer and processor files. See NOTICE and LICENSE for component attribution and license terms.

Explore the Vela model collection