trmteb-trnews
Turkish sentence-embedding model: trmteb/turkish-embedding-model
fine-tuned with MultipleNegativesRankingLoss on (title, abstract) pairs from
the TR-News corpus. This is a
retrieval fine-tune: it improves title↔abstract retrieval; broad topical
clustering is unchanged (the baseline already sits near the label-noise ceiling of
the TR-News news-desk labels).
An Apple-Silicon MLX float16 build (verified at parity) is also available as
trmteb-trnews-v1-mlx.
Training
- Base:
trmteb/turkish-embedding-model(BERTurk, 768-d, mean pooling) - Data: 100,000 filtered TR-News (title, abstract) pairs
- Loss: MultipleNegativesRankingLoss (in-batch negatives)
- Hyperparameters: batch 128, lr 2e-05, 1 epoch(s), bf16
- Max sequence length: 512 (the base model's capability; training truncated the short title/abstract pairs at 128 tokens, but the published model is not capped at 128)
Evaluation (held-out TR-News test + TrGLUE-STS)
Topic metrics use the full TR-News test split (n≈14k). topic kNN@1 / macro-kNN@1 / separation are held-within-noise gates, not optimisation targets.
| metric | baseline | fine-tuned |
|---|---|---|
| title→abstract R@1 | 0.8635 | 0.8745 |
| title→abstract MRR | 0.8989 | 0.9093 |
| topic kNN@1 | 0.7635 | 0.7624 |
| topic macro-kNN@1 | 0.5858 | 0.591 |
| topic separation | 0.0276 | 0.0315 |
| TrGLUE-STS spearman | 0.6686 | 0.6677 |
Benchmarks (Apple M4 Max, MLX / Metal)
Head-to-head vs multilingual embedders on Turkish news (compare_embedders.py,
one MLX process each). Separation = same−diff cluster cosine, which governs a
threshold-based dedup cut:
| metric | trnews-v1 (fp16) | base trmteb (fp32) | gemma-300m | granite-r2 | qwen3-0.6B |
|---|---|---|---|---|---|
| Clustering P@1 | 0.989 | 0.993 | 0.943 | 0.975 | 0.939 |
| Title→body R@1 | 0.983 | 0.983 | 0.967 | 0.967 | 0.933 |
| Separation | 0.655 | 0.661 | 0.170 | 0.120 | 0.345 |
| Speed (ms/title) | 0.79 | 0.84 | 1.30 | 1.13 | 5.97 |
| Peak GPU (MB) | 1713 | 2065 | 1338 | 1418 | 2015 |
The trmteb family holds a large separation lead — a stable ~0.35 dedup threshold vs a razor-thin, unstable ~0.80 for the multilingual models. The fp16 MLX build ≈ the fp32 base here, at lower memory and latency. (This clustering ground truth is trmteb-authored, so it is base-biased; the fine-tune's measured gain is the TR-News retrieval table above.)
Throughput (MLX/Metal vs PyTorch/MPS, full 12k-article corpus, same BERTurk arch): MLX ≈ 373 art/s, ~1.87× MPS (32 s vs 60 s), flat 2–3 GB peak — a full corpus re-embed ≈ 30 s.
Intended use & limitations
Turkish-news title↔abstract retrieval and same-event dedup. Broad topic classification, general STS, and sentiment are out-of-target — the fine-tune does not improve topical clustering over the base model; evaluate on your own task before relying on it.
License & attribution
Released under the MIT license — matching the verified upstream root
(dbmdz/bert-base-turkish-uncased, MIT).
NOTICE — this is a derivative work. Attribution chain:
- Fine-tuned from
trmteb/turkish-embedding-model— no license is declared upstream (as of 2026-07-03 the source model card states none); attribution retained. - Whose base is
dbmdz/bert-base-turkish-uncased(BERTurk) — MIT. Retain BERTurk's copyright and permission notice (see its model card). - Changes (documented for provenance): fine-tuned on 100,000 TR-News (title, abstract) pairs with MultipleNegativesRankingLoss; published at max_seq_length 512.
trmteb/turkish-embedding-model itself declares no license, so the MIT terms above
inherit from the BERTurk root and cover this fine-tune's contribution; confirm the
intermediate model's terms with its author before commercial redistribution.
Citation — training data (TR-News)
Please cite the TR-News dataset paper:
@article{10.1007/s10579-021-09568-y,
year = {2022},
title = {{Abstractive text summarization and new large-scale datasets for agglutinative languages Turkish and Hungarian}},
author = {Baykara, Batuhan and Güngör, Tunga},
journal = {Language Resources and Evaluation},
issn = {1574-020X},
doi = {10.1007/s10579-021-09568-y},
pages = {1--35}
}
TR-News license is unspecified upstream; this fine-tune cites the paper per the dataset authors' request. Confirm terms before commercial use.