ModernBERT-TR 1k
A Modern Encoder Foundation Model for Turkish
Besher Alkurdi, Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali
Web Page · Paper (soon) · Training Code · Evaluation Code
Overview
ModernBERT-TR 1k is a 150M-parameter Turkish encoder pretrained from scratch on 144.4B tokens using the ModernBERT architecture. It uses a custom 50K WordPiece tokenizer optimized for Turkish morphology.
Architecture: 22 layers, 768 hidden, 12 heads, RoPE, GLU, alternating local-global attention, Flash Attention, sequence-packed training.
Results
Frozen Linear Probing (11 Turkish NLP tasks)
| Model | Params | Avg |
|---|---|---|
| ModernBERT-TR 1k | 150M | 60.2 |
| Turkish-E5-large | 560M | 53.2 |
| mmBERT | 307M | 54.9 |
| TabiBERT | ~150M | 49.1 |
| BERTurk | 111M | 35.3 |
+13.1% relative over next-best. +70.3% relative over BERTurk. Outperforms models up to 4x larger.
TabiBench Full Fine-Tuning (28 tasks)
| Model | Params | Avg |
|---|---|---|
| ModernBERT-TR 1k | 150M | 77.28 |
| TabiBERT | ~150M | 77.58 |
| BERTurk | 110M | 75.96 |
Leads in 5/8 categories (text classification, STS, NLI, academic understanding, information retrieval). TabiBERT leads in code retrieval and QA (trained on code/math data).
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("ytu-ce-cosmos/modernbert-tr-base-1k")
model = AutoModelForMaskedLM.from_pretrained("ytu-ce-cosmos/modernbert-tr-base-1k")
text = "Türkiye'nin başkenti [MASK]'dır."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
Training Details
| Data | FineWeb-2 Turkish (41.2B tokens) + BertTurk Corpus 5x (31.0B tokens) = 72.2B/epoch, 2 epochs |
| Tokenizer | 50K WordPiece, trained on Turkish data |
| Optimizer | StableAdamW, peak LR 2e-4, cosine schedule |
| Batch size | 256 sequences (262K tokens/step) |
| MLM masking | 30% (train) / 15% (eval) |
| Hardware | 4x NVIDIA H100, 623 GPU-hours |
| Precision | BF16 mixed precision |
| Context | 1,024 tokens |
Citation
@article{modernberttr1k,
title = {Modern{BERT}-{TR}: A Modern Encoder Foundation Model for {Turkish}},
author = {Alkurdi, Besher and Kesgin, Himmet Toprak and Yuce, Muzaffer Kaan and Amasyali, Mehmet Fatih},
journal = {Research Square},
year = {2026},
doi = {10.21203/rs.3.rs-9500377/v1},
url = {https://doi.org/10.21203/rs.3.rs-9500377/v1},
note = {Preprint},
}
Acknowledgments
Supported by Yildiz Technical University (FDK-2024-6070) and TUBITAK (124E055). Built on the ModernBERT codebase with FineWeb-2 and BertTurk Corpus data.