VelaVec-T2I
Text→Image retrieval variant of VelaVec — a lightweight hybrid retrieval encoder (frozen distilled table + 3-layer bidirectional attention + dual pooling heads). This variant adds a vision projection head that maps CLIP teacher image features into the VelaVec text-embedding space, enabling cross-modal retrieval on top of the 9.8M-parameter text trunk.
| VelaVec-T2I | task | direction | |
|---|---|---|---|
| text→img r@1 (flickr30k) | 0.560 | caption → image retrieval | t2i |
| img→text r@1 (flickr30k) | 0.619 | image → caption retrieval | i2t |
| CLIP teacher (upper bound) | 0.585 / 0.800 | b32+l14 ensemble | t2i / i2t |
Dedicated to the text→image direction (best single-head t2i on flickr30k eval). For image→text, see VelaVec-MultiModal (i2t r@1 = 0.650).
Model details
- Text trunk: identical to VelaVec — frozen 30,522×256 static table → 3 BiBlocks (RMSNorm, RoPE, SwiGLU) → attention pooling, dual heads (
head=0symmetric,head=1retrieval). - Vision head (
config.vision): projection-only. Frozen CLIP teacher features —openai/clip-vit-base-patch32(512d) +openai/clip-vit-large-patch14(768d), L2-normalized and concatenated (1280d) — then a residual MLP (1280 → R768 → 256) into VelaVec space. - No CLIP weights are bundled: at inference, teacher features are produced by the HF transformers
CLIPModel(auto-downloaded from the configured repo ids, offline cache supported); only the small projection head (±1.5M params) is inmodel.safetensors. - Projection training: all-5-description InfoNCE (caption→image softmax) with cosine-annealing + residual-anchor init from the best linear projection. No regression terms (they hurt t2i discrimination across images). Training data: flickr30k.
Usage
from modeling_velavec import VelaVec
model = VelaVec.from_pretrained("xagent2025/VelaVec-T2I")
# Text: VelaVec head-1 embeddings (retrieval head)
caps = model.encode(["a man riding a bicycle down a street"], head=1)
# Image: CLIP teacher features → projection head
from PIL import Image
imgs = [Image.open("photo.jpg")]
z_img = model.encode_image(imgs) # [B,256] in the same space as text
score = z_img @ caps.T # caption→image ranking
Image preprocessing: resized/center-cropped to 224×224, CLIP normalization (same as teacher). No instruction prompting.
Files
| File | Description |
|---|---|
model.safetensors | PyTorch weights — text trunk + vision projection head |
config.json | Architecture config (vision.mode="proj", dual teacher) |
modeling_velavec.py | Standalone inference module (text + image) |
tokenizer.*, vocab.txt | bge-small tokenizer (BAAI/bge-small-en-v1.5, MIT) |
Relationship to other models
- VelaVec — pure-text (no vision).
- VelaVec-T2I — text→image optimized (this repo).
- VelaVec-MultiModal — image→text balanced head (w-mse regression, i2t 0.650 / t2i 0.532).
Limitations
- Dedicated to t2i; its i2t (0.619) trails the balanced variant.
- Image inference requires PyTorch + transformers (teacher models); the pure-Rust path covers text only.
- Rust engine (text path, 53.9× vs teacher, cargo build, MIT): xagent2025/velavec-rust-inference
- flickr30k eval is a small paired retrieval pool (1000 images × 5000 captions) — indicative, not a full multimodal benchmark (e.g., COCO/Flickr30k 1K test protocols).