Vela Base
Vela Base is a multilingual encoder foundation for specialized routing models.
307M parameters · Input capacity: 32,768 tokens, including special tokens.
Evaluation
Masked-token negative log-likelihood on the same multilingual development set, compared with the original mmBERT32K Base. Lower is better.
| Context length | Original mmBERT | Vela |
|---|---|---|
| 512 tokens · 156 windows | 1.441 | 1.387 |
| 8K tokens · 54 windows | 1.226 | 1.212 |
| 16K tokens · 54 windows | 1.206 | 1.168 |
| 32K tokens · 54 windows | 1.075 | 1.018 |
The 318 windows cover English, Chinese, German, French, Japanese and Arabic. Both models use FP32, identical fixed 15% masked-token targets, and complete inputs without truncation. This development set informed Vela checkpoint selection; it is not an independent test set.
Quick start
With PyTorch and Transformers 4.57.6 or 5.17.0:
from transformers import pipeline
model_id = "llm-semantic-router/Vela-1.0-Encoder-307M"
model = pipeline("fill-mask", model=model_id, device=-1)
print(model(f"The capital of France is {model.tokenizer.mask_token}."))