berturk2berturk-tr-normalizer

A fine-tuned encoder-decoder model, built from two BERTurk (dbmdz/bert-base-turkish-cased) checkpoints, that rewrites informal, misspelled, or dialectal Turkish sentences into standard written Turkish.

Typical inputs are the kind of text people actually type in a hurry: dropped diacritics, missing letters, colloquial verb contractions, ASCII-only typing, and regional/dialectal spelling. The model rewrites these into a clean, standard-Turkish sentence while keeping the original meaning and word order.

yarın buluşucaz mı bilmiyom          -> Yarın buluşacağız mı bilmiyorum
kardesim eve gelirken ekmek al       -> Kardeşim eve gelirken ekmek al
cok tesekkur ederim yardımınız için  -> Çok teşekkür ederim yardımınız için

Model architecture

This model is not a natively sequence-to-sequence architecture like T5 — BERTurk is an encoder-only model (it reads and represents text, it does not generate text). To use it for this task, it was converted into an encoder-decoder pair with Hugging Face's EncoderDecoderModel:

model = EncoderDecoderModel.from_encoder_decoder_pretrained(
    "dbmdz/bert-base-turkish-cased", "dbmdz/bert-base-turkish-cased"
)

• Encoder: BERTurk's pretrained weights, used as-is.
• Decoder: a second copy of BERTurk's weights, switched into decoder mode (causal self-attention).
• Cross-attention: the bridge that lets the decoder attend to the encoder's output. BERTurk does not have this by construction — these layers are randomly initialized and learned entirely from scratch during fine-tuning.
• Three generation-related config values that come for free with a native seq2seq model had to be set explicitly: decoder_start_token_id ([CLS]), eos_token_id ([SEP]), pad_token_id ([PAD]).

Two other differences from a byte-level seq2seq setup worth knowing:

• Tokenizer: BERTurk's own WordPiece tokenizer (32k vocabulary) rather than a byte-level tokenizer — severely garbled or out-of-vocabulary words can map to [UNK], which a byte-level model would not do.
• No task prefix: BERT's pretraining doesn't use T5-style task prefixes, so the input is passed to the model as-is, with no prepended instruction string.

How to use

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "erdemKocaogluu/berturk2berturk-tr-normalizer"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

text = "yarın buluşucaz mı bilmiyom"
inputs = tokenizer(text, return_tensors="pt")
output = model.generate(**inputs, max_length=128, num_beams=4)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training data

Training pairs come from six public Turkish datasets that already provide noisy/clean sentence pairs (grammar correction, typo validation, text normalization — see Data sources below). No data was synthetically generated for this training run. Four sources are permissively licensed (MIT, Apache-2.0, or CC0-1.0); two are derived from Turkish Wikipedia and are licensed CC BY-SA 3.0 (attribution required — see below).

Data sources

Not every dataset that was evaluated made it into the final training mix — a diacritics-restoration dataset, for example, was tested but excluded after evaluation.

The MIT, Apache-2.0, and CC0-1.0 sources above are acknowledged here per their license terms; their original copyright notices and licenses remain with the linked repositories. The trspell (NoisyWikiTr) and noisy-sentences-dataset sources are built from Turkish Wikipedia text; that text is licensed CC BY-SA 3.0, so it is credited here accordingly: portions derived from Wikipedia contributors, licensed under CC BY-SA 3.0. This project does not redistribute the original dataset files — only sentence pairs derived from them were used to train the model.

Note on Turkish-OSCAR-GEC: this dataset is built on top of OSCAR, a Common Crawl–derived web text corpus. OSCAR's own documentation states that its maintainers "do not own any of the text from which these data has been extracted" and license only their own metadata/annotations (CC0). The "MIT" tag on Turkish-OSCAR-GEC is the uploader's own declaration and does not itself establish clear rights over the underlying crawled text. This is disclosed here so downstream users can make their own informed assessment; it reflects a broader, industry-wide open question around web-scraped training corpora rather than an issue specific to this dataset.

OSCAR's own terms note an exception for text-and-data-mining (TDM) and research use; the way this data was used here — extracting statistical spelling/grammar-correction patterns during training, not reproducing or redistributing OSCAR's source sentences — fits that framing. The model is also not a free-text generator: it transforms a sentence the user supplies, so its output is structurally anchored to that input rather than free to reproduce memorized training text, which limits (though does not eliminate) the risk of verbatim reproduction of training data.

Training procedure

During training, checkpoints were evaluated periodically on a held-out validation split. Each checkpoint is scored with:

Q_good         = 1 - (0.6 * CER + 0.4 * WER)      # how well it fixes noisy sentences
unchanged_rate = pred == source on clean sentences  # how well it leaves correct text alone
selection_score = 0.7 * Q_good + 0.3 * unchanged_rate

The checkpoint used here is the highest-scoring one from that process (selection_score 0.9907, out of 283 evaluated checkpoints over 566,026 total steps), and its weights were verified by hash before being published.

Training curves

Evaluation

Held-out gold sets (naturally occurring, human-written Turkish — never seen during training)

The model roughly halves the word error rate on the tweets set compared to a "do nothing" baseline. On the boun set (a de/da conjunction–focused benchmark), it corrects real errors but at a higher over-correction cost than on tweets — see Limitations below.

Sources:

• tweets — Köksal, A. T., Bozal, Ö., Yürekli, E., & Gezici, G. (2020). #Turki$hTweets: A Benchmark Dataset for Turkish Text Correction. Findings of ACL: EMNLP 2020, pages 4190–4198.
• boun — Kara, A., Marouf Sofian, F., Bond, A., & Şahin, G. (2023). GECTurk: Grammatical Error Correction and Detection Dataset for Turkish. Findings of ACL: IJCNLP-AACL 2023, pages 278–290.

The tweets set was lightly cleaned before evaluation — social-media artifacts (@mentions, hashtags, emoji) were stripped from both source and reference, and rows the original dataset flagged as foreign-language or neologism were excluded.

Limitations

• The model generalizes very well to mechanical, deterministic corrections (like deasciification), where the rule mapping is unambiguous.
• It is comparatively weaker on organic, human-style noise that doesn't closely match the synthetic noise patterns it was trained on, and on long, multi-clause informal sentences.
• It corrects spelling, spacing, and word-internal errors; it does not reorder words or rewrite sentence structure.
• The WordPiece tokenizer can map severely garbled or out-of-vocabulary words to [UNK], losing information a byte-level tokenizer would preserve.
• Maximum input/output length is 128 tokens (shorter than a byte-level model's typical window at the same sequence length), so very long sentences get truncated.
• The cross-attention bridge between encoder and decoder was learned entirely from scratch during fine-tuning (BERTurk has no such mechanism natively); this makes the model more prone to occasional under-correction on inputs that differ structurally from training data.

License

These fine-tuned model weights are released under CC BY-NC-SA 4.0 (non-commercial use, attribution required, share-alike). Note this differs from the base model: dbmdz/bert-base-turkish-cased itself remains available under its own MIT license. This license applies to how these fine-tuned weights are released; it is separate from the licenses of the training data sources listed above under Data sources, consistent with common practice for models trained on mixed-license corpora.