A 150M-parameter Turkish cross-encoder reranker to score (query, document) relevance.
- Base model:
ytu-ce-cosmos/modernbert-tr-base. - Distilled from
Qwen/Qwen3-Reranker-8B.
Results
Reranking the top-100 of a first-stage retriever (ytu-ce-cosmos/modernbert-tr-embed) at max_seq=512. The uplift (Δ) is the reranker's contribution.
| Task | First-stage NDCG@10 | + Reranker | Δ |
|---|---|---|---|
| ArguAnaTR | 37.01 | 54.75 | +17.74 |
| SquadTRRetrieval | 75.94 | 90.11 | +14.17 |
| SciFactTR | 77.07 | 86.34 | +9.27 |
| TQuadRetrieval | 87.48 | 94.00 | +6.52 |
| CQADupstackGamingRetrievalTR | 56.44 | 61.10 | +4.66 |
| XQuADRetrieval | 95.03 | 97.86 | +2.83 |
| Mean Δ | +9.20 |
How was this model trained?
Question answering and counter argument distillation of Qwen3-Reranker-8B relevance scores into the 150M cross-encoder over Turkish question answering / information retrieval data using listwise KL.
Usage
transformers
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker")
model = AutoModelForSequenceClassification.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker").eval()
query = "Türkiye'nin başkenti neresidir?"
docs = ["Ankara, Türkiye'nin başkentidir.", "İstanbul en kalabalık şehirdir."]
enc = tok([query] * len(docs), docs, padding=True, truncation="longest_first",
max_length=8192, return_tensors="pt")
with torch.no_grad():
scores = model(**enc).logits.squeeze(-1)
ranking = sorted(zip(docs, scores.tolist()), key=lambda x: x[1], reverse=True)
sentence-transformers
from sentence_transformers import CrossEncoder
model = CrossEncoder("ytu-ce-cosmos/modernbert-tr-reranker")
scores = model.predict([(query, d) for d in docs])
ONNX Runtime
The onnx/ folder has the full graph, the output is the relevance logit:
import onnxruntime, numpy as np
sess = onnxruntime.InferenceSession("onnx/model.onnx")
feed = {k: v.numpy() for k, v in enc.items() if k in {i.name for i in sess.get_inputs()}}
logits = sess.run(None, feed)[0].squeeze(-1)
Text Embeddings Inference (TEI)
text-embeddings-router --model-id ytu-ce-cosmos/modernbert-tr-reranker --dtype float16
# POST /rerank {"query": "soru", "texts": ["aday 1", "aday 2"]}
Training data
We used Turkish datasets msmarco-tr, squad-tr, fiqa-tr, nfcorpus-tr, quora-tr, scifact-tr for distillation by Qwen3-Reranker-8B, and Turkish counter-argument pairs from ArguAna machine-translated with TranslateGemma-27B. All training data was text-hash chceked against every MTEB(Turkish) test split.
Limitations
- Reported NDCG is rerank-of-top-100 over a first-stage retriever; absolute scores depend on that first stage.
- int8 ONNX reorders scores meaningfully lossy for a reranker; use fp32 for quality-sensitive ranking.
- Due to the lack of long form data in our training, the model's performance may degrade on long context input.
License & attribution
- License:
apache-2.0.