Vela Omni Nano
Text, images, speech and environmental audio in one embedding space. Vela Omni Nano supports multimodal search, routing and clustering with normalized vectors that can be compared directly.
Try Vela Studio · Vela collection · Detailed evaluation
Model overview
| Feature | Value |
|---|---|
| Modalities | Text, images, speech and environmental audio |
| Total parameters | 163.8M (163,771,288) |
| Embedding dimensions | 384 |
| Text context | 512 tokens, including special tokens |
| Audio input | Original PCM at 16, 44.1 or 48 kHz; up to 30 seconds |
| Output | L2-normalized vectors; cosine similarity |
| License | Apache 2.0; see component attribution |
A frozen CLAP audio branch adds environmental-sound information to the existing speech representation. Text and image computations are retained; audio embeddings are newly trained and evaluated. Architecture and measured identity.
Evaluation
Scores are 0–100; higher is better. The comparison uses the same held-out evaluation examples, complete retrieval pools and 128-token text cap for both models. Bold marks improvement over the original small model. These known test pools are reused across releases.
| Metric | Original small | Current Nano |
|---|---|---|
| Banking77 · Accuracy | 70.42 | 87.99 |
| MASSIVE English · Accuracy | 65.95 | 81.97 |
| COCO · Image → text · R@1 | 40.83 | 60.87 |
| COCO · Text → image · R@1 | 30.18 | 55.82 |
| LibriSpeech · Audio → text · R@1 | 4.21 | 16.12 |
| LibriSpeech · Text → audio · R@1 | 9.58 | 20.34 |
The common protocol uses labeled TRAIN prototypes for text classification and all matching positives for retrieval; it is separate from official MTEB classification. All 14 metrics, Macro-F1, exact counts and uncertainty.
Standard benchmarks and rankings
The primary metric, Mean(TaskType), weights each task type equally. Mean(Task) weights individual tasks equally and is supplementary. Rankings use complete benchmark results from the September 17, 2026 registry snapshots plus both current Vela models for English/audio and Nano for multilingual; “≤ size” counts models with no more total parameters than this model.
| Benchmark | Mean(TaskType) | Global rank | Rank at ≤ size | Gap to best at ≤ size | Mean(Task) |
|---|---|---|---|---|---|
| MTEB English v2 · 41 tasks | 60.78 | 66/188 | 5/75 | 0.61 pp | 64.88 |
| MAEB audio-only · 19 tasks | 52.34 | 19/64 | 6/27 | 3.51 pp | 43.59 |
| MTEB Multilingual v2 · 131 tasks | 34.01 | 81/89 | 21/26 | 17.11 pp | 42.55 |
Multilingual coverage remains a limitation. All 131 task scores and protocol.
Nano ranks 5/75 on the English panel and 6/27 on the audio panel among models with no more than its 163.8M total parameters. These are snapshot-relative comparisons across reported protocols, including single-modality specialists. The original small has not been evaluated on these complete panels. Full rankings and both aggregate metrics · All task scores and methods.
Audio Mean(TaskType) improves from 46.97 to 52.34 over the previous Nano. Parameters rise 20.97%; speech–text retrieval and VehicleSoundClustering regress. Complete gains and trade-offs.
Quality and model size
Each plot combines the full benchmark score, global and size-constrained ranks, and the gap to the best model at no greater total size. Every complete model with a known size is plotted, including models below the observed Pareto frontier. Highlighting Vela does not imply frontier membership. Click either figure for full resolution. The English panel uses Nano's default shared text and Mini's optional official-instruction mode; audio uses the default mode for both.
Selected task strengths
These task-level comparisons highlight specific strengths; they do not establish overall benchmark SOTA.
Comparison data and methodology · Figure gallery
Usage
Use PyTorch and matching torchaudio, Transformers 4.57.6, Hugging Face Hub, safetensors, NumPy, and Pillow:
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("llm-semantic-router/Vela-1.0-Omni-Nano")
sys.path.insert(0, path)
from vela_omni import VelaOmni
model = VelaOmni.from_pretrained(path, device="cpu")
vectors = model.encode_text(["A bicycle beside a building.", "Someone is reading aloud."])
print(vectors.shape) # (2, 384)
Text inputs support up to 512 tokens, including special tokens. Longer inputs raise ValueError; shorten or explicitly chunk them.
Pass a list of Pillow images to model.encode_image(images). Pass original NumPy waveforms to model.encode_audio(waveforms, sampling_rate=48000) using their actual 16,000, 44,100 or 48,000 Hz rate; each waveform must be at most 30 seconds. Mono or channels-first arrays are supported. Keep the original waveform: the speech and CLAP branches independently derive their 16 kHz and 48 kHz inputs. Do not downsample to 16 kHz before calling the API when higher-rate PCM is available. Compare normalized vectors with their dot product. To route media, embed each destination’s name and description, then select the closest vector. To discover routing categories, cluster media embeddings and inspect each group. Similarity scores are rankings, not calibrated probabilities.
Training and license
Cross-modal alignment uses COCO image–caption pairs from the CC-BY 2.0 image subset and LibriSpeech speech–transcript pairs (CC-BY 4.0). COCO annotations are CC-BY 4.0. The frozen GIST text backbone is MIT-licensed; its pinned model card and component notices are included.
Audio residual alignment uses 3,299 FSD50K TRAIN recordings under CC0 or CC-BY 3.0; per-recording attribution is included. The frozen CLAP unfused audio component is Apache 2.0; its original license is retained as LICENSE-CLAP.
The repository includes the native model code, component configurations, and tokenizer and processor files. See NOTICE and LICENSE for component attribution and license terms.