BERTurk-128k Turkish News Classification (9-class)

Fine-tuned dbmdz/bert-base-turkish-128k-cased (BERTurk, 128k cased vocabulary) for single-label, multiclass topic classification of Turkish news text into 9 categories.

The model was trained on the TRNEWS-2025 dataset (a large, real-world, and highly imbalanced Turkish news corpus) with class-balanced cross-entropy loss. On a held-out, stratified test split of 40,063 documents it reaches 95.95% accuracy, 0.9602 weighted-F1, and 0.9357 macro-F1.

• Developed by: thealper2
• Model type: BertForSequenceClassification (encoder-only Transformer)
• Language: Turkish (tr)
• Finetuned from: dbmdz/bert-base-turkish-128k-cased
• License: MIT (inherited from the BERTurk base model; see Licensing for dataset terms)
• Task: Multiclass text classification (topic / news-category)

Labels

The classifier outputs one of 9 mutually exclusive news categories. Labels are stored in the model config (id2label / label2id) in their original Turkish surface forms; English glosses are provided below for convenience.

Because id2label is baked into the config, pipeline(...) and model.config.id2label return the human-readable Turkish label directly — no external mapping file is required at inference time.

Quick start

With pipeline

from transformers import pipeline

clf = pipeline(
    "text-classification",
    model="thealper2/berturk-128k-turkish-news-classification",
    truncation=True,
    max_length=512,
)

clf("Milli takım, hazırlık maçında sahadan galibiyetle ayrıldı.")
# [{'label': 'Spor', 'score': 0.99...}]

# Full probability distribution over all 9 classes:
clf("Borsa İstanbul günü yükselişle kapattı, faiz beklentileri fiyatlandı.",
    top_k=None)

With AutoModel (manual, batched)

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "thealper2/berturk-128k-turkish-news-classification"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()

texts = [
    "Yeni sürücüsüz araç teknolojisi trafik güvenliğini artırmayı hedefliyor.",
    "Antalya'da turizm sezonu erken açıldı, oteller yüksek doluluğa ulaştı.",
]

enc = tokenizer(texts, padding=True, truncation=True,
                max_length=512, return_tensors="pt")

with torch.no_grad():
    logits = model(**enc).logits
    probs = logits.softmax(dim=-1)

pred_ids = probs.argmax(dim=-1).tolist()
for text, pid, p in zip(texts, pred_ids, probs):
    print(f"{model.config.id2label[pid]:16s} ({p[pid]:.3f})  <- {text[:50]}")

Tip: the model was trained on article bodies (Haber Gövdesi) rather than headlines, so it performs best on full paragraphs/articles. Very short headlines may be classified less reliably. Always pass truncation=True, max_length=512.

Intended uses & limitations

Intended uses

• Automatic topic tagging / routing of Turkish news articles.
• Editorial content organization, feed categorization, and dataset labeling assistance.
• A strong Turkish-language baseline / backbone for downstream topic classification.

Out-of-scope / limitations

• Domain shift: trained on news text. Performance will degrade on other genres (tweets, product reviews, legal/medical documents, conversational text).
• Single-label only: the head is a softmax classifier — it forces exactly one category even for articles that legitimately span multiple topics (e.g. sports-economy). It cannot emit multiple labels.
• Label imbalance: the corpus is dominated by Magazin (~39% of the data). Even though class-balanced loss was used, the tail classes (Çevre, Turizm, Bilim-Teknoloji, Kültür-Sanat) show lower precision and are more prone to confusion (see the confusion analysis).
• No factuality/sentiment signal: the model predicts topic, not veracity, sentiment, or stance.
• Potential bias: the model reflects the editorial and demographic biases of the source Turkish news outlets in TRNEWS-2025.

Training data

• Dataset: TRNEWS-2025: A Multiclass Turkish News Text Dataset (IEEE DataPort, DOI 10.21227/vasv-dj22, Sengul Bayrak, 2025). Access requires an IEEE DataPort subscription.
• Input column: Haber Gövdesi (news body). Label column: Sınıf (class). Auxiliary columns (ID, Erişim Tarihi, URL) were dropped.

Preprocessing

Applied before tokenization (dataset-agnostic cleaning pipeline):

• HTML unescaping + tag stripping.
• URL stripping (http(s)://…, www.…).
• Whitespace normalization.
• Casing preserved (lowercase: false) — the base model is cased; no Turkish İ/ı folding.
• Exact-duplicate removal on the cleaned text.
• Empty/too-short rows removed (min_chars: 1).

Splits

Stratified train/validation/test split (seed=42), 90% / 5% / 5%:

Class distribution (full corpus, approximate)

The dataset is heavily imbalanced (≈20:1 between the largest and smallest class), which motivated class-weighted training.

(Counts are inferred from the stratified test-split support × 20; they are accurate to within rounding.)

Training procedure

Fine-tuned end-to-end with the HuggingFace Trainer (a custom WeightedTrainer subclass applying per-class weights to the cross-entropy loss).

Class-balanced loss

Cross-entropy is weighted with the scikit-learn "balanced" scheme, computed on the training split:

weight_c = N_train / (num_classes * count_c)

This yields weights ranging from ≈ 0.28 for the majority class (Magazin) to ≈ 5.7 for the minority class (Çevre), pulling macro-F1 up substantially relative to unweighted training.

Hyperparameters

Compute

• Hardware: single NVIDIA RTX 5060 Ti (Blackwell, sm_120, 8 GB VRAM).
• Frameworks: PyTorch (CUDA 12.8 wheel) · Transformers 4.57.6.
• 8 GB VRAM was made feasible through bf16, SDPA attention, dynamic padding, and length grouping (fixed 512-token padding was avoided).

Convergence & early stopping

Because the training set is large (~721k examples), the model converged within a fraction of the first epoch. Validation f1_macro peaked at step 3000 and did not improve over the next two evaluations, so EarlyStoppingCallback halted training at step 4000 and restored the best (step-3000) checkpoint. Selected validation metrics:

Note: the low epoch count is expected, not a bug — a single epoch over ~721k documents is a large number of optimization steps. Training longer offered no macro-F1 gain and risked overfitting the majority class.

Evaluation

All metrics below are on the held-out test split (40,063 documents), which was never seen during training or model selection.

Overall

Per-class report

Confusion analysis

Rows = true label, columns = predicted label (order matches the label table above).

Key takeaways:

• High-support, well-separated classes (Magazin, Spor, Siyaset, Finans-Ekonomi) exceed 0.96 F1.
• The main error mode is precision loss on the minority classes Kültür-Sanat (0.82) and Bilim-Teknoloji (0.84): a slice of Magazin articles leaks into Kültür-Sanat (279 cases), and several Siyaset articles are pulled into Bilim-Teknoloji. These are semantically adjacent topics in Turkish news.
• Class weighting keeps recall high across every class (min. recall 0.92), which is often the desired trade-off for tail categories.

Reproducing evaluation

The exact metrics, full classification_report, and confusion matrix are shipped in test_report.json alongside these weights.

Model architecture

Licensing

• Model weights: released under the MIT License, inherited from the base model dbmdz/bert-base-turkish-128k-cased.
• Training data (TRNEWS-2025): distributed via IEEE DataPort and requires a subscription to access. This repository ships only the fine-tuned weights and evaluation artifacts — no raw dataset text is included or redistributed. If you use the data itself, you must comply with IEEE DataPort's terms and cite the dataset (below).

Citation

If you use this model, please cite both the underlying dataset and the base model.

Dataset (TRNEWS-2025):

@data{vasv-dj22-25,
  doi       = {10.21227/vasv-dj22},
  url       = {https://dx.doi.org/10.21227/vasv-dj22},
  author    = {Sengul Bayrak},
  publisher = {IEEE Dataport},
  title     = {TRNEWS-2025: A Multiclass Turkish News Text Dataset},
  year      = {2025}
}

Base model (BERTurk):

@software{stefan_schweter_2020_3770924,
  author    = {Stefan Schweter},
  title     = {BERTurk - BERT models for Turkish},
  month     = apr,
  year      = 2020,
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.3770924},
  url       = {https://doi.org/10.5281/zenodo.3770924}
}

This fine-tuned model:

@misc{berturk_128k_turkish_news_classification,
  author    = {thealper2},
  title     = {BERTurk-128k Turkish News Classification (9-class)},
  year      = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/thealper2/berturk-128k-turkish-news-classification}}
}

Acknowledgements

• Sengul Bayrak (Istanbul Sabahattin Zaim University) for the TRNEWS-2025 dataset.
• Stefan Schweter / dbmdz for the BERTurk base models.
• The Hugging Face transformers, datasets, and accelerate teams.