LocRet-small — Azerbaijani Retrieval Embedding Model

LocRet-small is a compact, high-performance retrieval embedding model specialized for the Azerbaijani language. Despite being 4.8× smaller than BGE-m3, it significantly outperforms it on Azerbaijani retrieval benchmarks.

Key Results

AZ-MIRAGE Benchmark (Native Azerbaijani Retrieval)

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("LocalDoc/LocRet-small")

queries = ["Azərbaycanın paytaxtı hansı şəhərdir?"]
passages = [
    "Bakı Azərbaycan Respublikasının paytaxtı və ən böyük şəhəridir.",
    "Gəncə Azərbaycanın ikinci böyük şəhəridir.",
]

query_embeddings = model.encode_query(queries)
passage_embeddings = model.encode_document(passages)

similarities = model.similarity(query_embeddings, passage_embeddings)
print(similarities)

Prefixes "query: " and "passage: " are applied automatically via encode_query and encode_document. If using model.encode directly, the "passage: " prefix is added by default.

Training

Method

LocRet-small is fine-tuned from multilingual-e5-small using listwise KL distillation combined with a contrastive loss:

$$\mathcal{L} = \mathcal{L}{\text{KL}} + 0.1 \cdot \mathcal{L}{\text{InfoNCE}}$$

• Listwise KL divergence: Distills the ranking distribution from a cross-encoder teacher (bge-reranker-v2-m3) over candidate lists of 1 positive + up to 10 hard negatives per query. Teacher and student softmax distributions use asymmetric temperatures (τ_teacher = 0.3, τ_student = 0.05).
• In-batch contrastive loss (InfoNCE): Provides additional diversity through in-batch negatives on positive passages.

This approach preserves the full teacher ranking signal rather than reducing it to binary relevance labels, which is critical for training on top of already strong pre-trained retrievers.

Data

The model was trained on approximately 3.5 million Azerbaijani query-passage pairs from four datasets:

All datasets include hard negatives scored by a cross-encoder reranker, which serve as the teacher signal for listwise distillation. False negatives were filtered using normalized score thresholds.

Hyperparameters

Training Insights

• Listwise KL distillation outperforms standard contrastive training (MultipleNegativesRankingLoss) for fine-tuning pre-trained retrievers, consistent with findings from Arctic-Embed 2.0 and cadet-embed.
• Retrieval pre-training matters more than language-specific pre-training for retrieval tasks: multilingual-e5-small (with retrieval pre-training) significantly outperforms XLM-RoBERTa and other BERT variants (without retrieval pre-training) as a base model.
• A mix of translated and native data prevents catastrophic forgetting while enabling language specialization.

Benchmark

AZ-MIRAGE

A native Azerbaijani retrieval benchmark (https://github.com/LocalDoc-Azerbaijan/AZ-MIRAGE) with 7,373 queries and 40,448 document chunks covering diverse topics. Evaluates retrieval quality on naturally written Azerbaijani text.

Model Details

Limitations

• Optimized for Azerbaijani text retrieval. Performance on other languages may be lower than the base multilingual-e5-small model.
• Maximum input length is 512 tokens. Longer documents should be chunked.

Citation

@misc{locret-small-2026,
  title={LocRet-small: A Compact Azerbaijani Retrieval Embedding Model},
  author={LocalDoc},
  year={2026},
  url={https://huggingface.co/LocalDoc/LocRet-small}
}

Acknowledgments

• Base model: intfloat/multilingual-e5-small
• Teacher reranker: BAAI/bge-reranker-v2-m3
• Training methodology inspired by Arctic-Embed 2.0 and cross-encoder listwise distillation research.