Loss-Guided Static Multilingual Sentence Embeddings
Overview
A static multilingual sentence encoder: one embedding table, mean pooling, no Transformer at inference. Its vocabulary was extended with loss-guided vocabulary discovery: during contrastive training, adjacent token pairs are scored by how much the sentence objective would suffer without them, and the useful pairs receive their own embedding rows. The base tokenizer stays unchanged. The model is trained on 176 languages and is intended for CPU-efficient semantic similarity, retrieval, clustering, and classification.
Model Specifications
- Embedding dimension: 1,024
- Base vocabulary: 257,914 rows (mmBERT tokenizer + 1,914 language markers)
- Learned pair rows: 56,071
- Parameters: 321.7M (embedding table only), FP32 safetensors, 1.29 GB
- Training languages: 176
- Inference: table lookup and mean pooling; requires a language marker at the start of each text
Performance
MTEB(Multilingual, v2), 131 tasks, primary score (Mean over tasks) 50.13, Mean over task types 43.45:
| Task type | Count | Mean score |
|---|---|---|
| Bitext mining | 13 | 53.90 |
| Classification | 43 | 52.87 |
| Clustering | 16 | 38.91 |
| Instruction reranking | 3 | -1.04 |
| Multilabel classification | 5 | 17.64 |
| Pair classification | 11 | 72.69 |
| Reranking | 6 | 49.42 |
| Retrieval | 18 | 41.64 |
| STS | 16 | 64.99 |
Cross-lingual retrieval (macro F1 over language pairs): Tatoeba 57.26 (106 pairs), BUCC 98.12 (4 pairs), Flores 63.70 (193 pairs).
Installation & Usage
Every input must start with a language marker __<iso639-3>_<script>__, for example __eng_Latn__, __rus_Cyrl__, __arb_Arab__, __cmn_Hani__. A text without a marker raises an error. The repository ships the custom embedding module, so trust_remote_code=True is required.
pip install -U sentence-transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BorisTM/loss-guided-static-multi", trust_remote_code=True)
sentences = [
"__eng_Latn__ The cat sleeps on the sofa.",
"__rus_Cyrl__ Кошка спит на диване.",
"__deu_Latn__ Die Katze schläft auf dem Sofa.",
"__eng_Latn__ The stock market fell sharply today.",
]
embeddings = model.encode(sentences, normalize_embeddings=True)
similarities = model.similarity(embeddings, embeddings)
# [[1.00, 0.44, 0.80, 0.05],
# [0.44, 1.00, 0.41, 0.06],
# [0.80, 0.41, 1.00, 0.08],
# [0.05, 0.06, 0.08, 1.00]]
License: Apache 2.0