darmm-embed-kazakh-v2
General-purpose Kazakh retrieval embedding model — semantic search, RAG, deduplication, and clustering for Kazakh text, with strong cross-lingual alignment to Russian and English (a Kazakh query finds Russian/English documents by meaning). Supersedes darmm-embedding-multilingual.
Fine-tuned from BAAI/bge-m3 (568M, 1024-dim, up to 8k tokens) on 373k Kazakh pairs from Darmm/darmm-kk-retrieval-pairs: Kazakh Wikipedia structural pairs (title↔lead, section, adjacent paragraphs), kk↔ru (WikiMatrix) and kk↔en (NLLB) parallel sentences, and Kazakh instruction QA pairs.
Evaluation
Held-out sets from the pairs dataset (never trained on). Retrieval: 6,322 article titles as queries against 6,322 lead paragraphs. Cross-lingual: match a Kazakh sentence to its translation in a 2,000-candidate pool.
| metric | this model | BAAI/bge-m3 | darmm-embedding-multilingual (v1) |
|---|---|---|---|
| kk retrieval Recall@1 | 0.716 | 0.666 | 0.662 |
| kk retrieval Recall@5 | 0.754 | 0.714 | 0.705 |
| kk retrieval Recall@10 | 0.775 | 0.730 | 0.721 |
| kk retrieval MRR@10 | 0.733 | 0.687 | 0.681 |
| kk→ru Accuracy@1 | 0.668 | 0.518 | 0.546 |
| kk→en Accuracy@1 | 0.918 | 0.851 | 0.855 |
Reproducible with eval_embed.py in this repo.
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Darmm/darmm-embed-kazakh-v2")
query = "Астанада ауа райы қандай?"
docs = [
"Елордада бүгін күн ашық, +25 градус болады.",
"Погода в столице сегодня солнечная.",
"Доллар бағамы тағы өзгерді.",
]
q_emb = model.encode([query], normalize_embeddings=True)
d_emb = model.encode(docs, normalize_embeddings=True)
print((q_emb @ d_emb.T)) # cosine similarities
Training
MultipleNegativesRankingLoss (in-batch negatives, no-duplicates sampler), 1 epoch over 373k pairs, batch 32, max_seq 192, lr 2e-5 cosine with 5% warmup, bf16, single A100. Final train loss 0.137.
Limitations
- Trained with 192-token pairs; quality on much longer passages is inherited from bge-m3 rather than tuned.
- kk↔ru pairs come from mined alignments (WikiMatrix) and carry some noise.
- Domain-specific jargon (legal, medical) is underrepresented in Wikipedia-based training data.
Built by Darmm · 2026-09 · Apache 2.0