VelaVec

A lightweight hybrid retrieval encoder: frozen distilled table + 3 layers of bidirectional attention + dual pooling heads.

Built for latency-sensitive retrieval on edge hardware: a pure-Rust inference path (BNNS/CoreML + NEON kernels) with no Python runtime.

Model details

• Trunk: frozen 30,522×256 static table (distilled from bge-small token embeddings + PCA) → 3 transformer blocks (RMSNorm, RoPE, SwiGLU FFN) → attention pooling.
• Dual heads: head=0 symmetric semantics (STS / similarity), head=1 asymmetric retrieval (query ↔ doc). One trunk, two uses.
• Training: 5-domain distillation (NLI + STS + MS MARCO retrieval + SciFact/NFCorpus/ArguAna) with teacher-embedding regression (relational + InfoNCE). See training notes.

Evaluation (MTEB, no prompts, max_len=256, CPU)

Domain-internal retrieval (own eval pools): NLI r@1 = 0.912, MS MARCO dev r@1 = 0.694 (1k-candidate pool).

Usage

Python (no transformers dependency for inference)

from modeling_velavec import VelaVec

model = VelaVec.from_pretrained("velavec_hf")   # or a HF repo id

# Symmetric semantics (STS / clustering) — head 0
sim = model.encode(["a man is playing guitar on stage",
                    "a child reads a book under a tree"], head=0)

# Asymmetric retrieval (query vs passage) — head 1
q = model.encode(["what is the capital of france"], head=1)
docs = model.encode(["Paris is the capital of France and its largest city."], head=1)
score = q @ docs.T

Tokenization uses BAAI/bge-small-en-v1.5 (bundled under tokenizer.* / vocab.txt).

Rust (edge inference)

weights.bin (39 MB, fp32, dual-head layout) is consumed by the Rust inference engine — BNNS/CoreML pre-packed GEMM graphs + NEON kernels, no Python/PyTorch at runtime.

Source code (cargo build; runs NLI+MARCO eval and latency benchmarks out of the box): xagent2025/velavec-rust-inference

• Query (len=15): 103 µs / 53.9× faster than teacher on CPU
• Doc (len=128): ~580 µs dual-threaded / ~780 µs single-threaded
• Cold start 5.4 ms; NLI throughput ~11.5k items/s, MARCO passages ~2.9k/s (2 threads)
• Byte-identical to PyTorch forward pass (max |Δ| = 0)

Head routing: NLI/similarity → head 0, MARCO retrieval → head 1.

Limitations

• OOD retrieval domains (medical / scientific / debate) reach 73–88% of teacher — the frozen 9.8M trunk can't fully absorb the 33M teacher's domain knowledge.
• Short input focus: best with ≤64 tokens for queries; docs up to 256 tokens.
• No instruction prompting (models trained without prompts; use as-is).

Training notes

Distillation protocol: teacher bge-small embeds → static table (PCA 256) → hybrid trunk warm-started from an earlier retrieval run → 5-domain multi-task distillation (relational KD + InfoNCE + embedding regression, symmetric & retrieval heads). Training data includes MS MARCO (research license) — review your use case for commercial compliance.

Files