KOZTAM — Koklear Okuma Zorluğu Tahminleme Modeli

KOZTAM (Cochlear Reading-Difficulty Forecasting Model) grades a Turkish text on an eight-level reading-difficulty scale (grades 1–8) with a frozen BERTurk encoder and a lightweight CORAL ordinal head, reaching 0.33 ordinal MAE, 0.72 exact-grade accuracy, and 0.96 within-one-grade accuracy on held-out test data.

Data

KOZTAM is trained on the behAIvNET/KOZTAM dataset — 1,588 synthetic Turkish texts spanning grades 1–8 in two categories (informative / narrative), generated under strict MEB-aligned readability targets. One exact-duplicate text (present under both a grade-3 and a grade-5 folder) was removed to eliminate a conflicting ordinal label, giving the final 1,588.

The texts are partitioned into 1,110 training / 239 validation / 239 test by two-stage stratified sampling over the 16 grade × category strata, so both the ordinal grade distribution and the ≈ 50/50 category balance are preserved in every split. Because the BERTurk backbone is frozen, each text's [CLS] embedding (768-d) is pre-computed once and cached, and only the head is trained on those cached vectors; the mean [CLS] norm is ≈ 25.2 and identical across the three splits, indicating no distributional shift between them.

A structural property worth noting for modelling: realized text length rises from grade 1 through grade 7 but drops at grade 8 (shorter, in both categories, than grades 5–7). Length is therefore not a monotone proxy for reading grade at the top of the scale, so the model must rely on lexical and syntactic density rather than on length.

Model

import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModel

DEVICE     = "cuda" if torch.cuda.is_available() else "cpu"
MODEL_NAME = "dbmdz/bert-base-turkish-cased"
MAX_LEN    = 512
K          = 8                     

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
bert = AutoModel.from_pretrained(MODEL_NAME).to(DEVICE).eval()
for p in bert.parameters():
    p.requires_grad_(False)

@torch.no_grad()
def encode_cls(texts):
    enc = tokenizer(texts, padding=True, truncation=True,
                    max_length=MAX_LEN, return_tensors="pt").to(DEVICE)
    return bert(**enc).last_hidden_state[:, 0, :]

class CoralHead(nn.Module):
    def __init__(self, in_dim, num_classes):
        super().__init__()
        self.fc   = nn.Linear(in_dim, 1, bias=False)
        self.bias = nn.Parameter(torch.zeros(num_classes - 1))
    def forward(self, x):
        return self.fc(x) + self.bias

class KOZTAMNet(nn.Module):
    def __init__(self, in_dim=768, p=0.2):
        super().__init__()
        self.proj = nn.Sequential(
            nn.Linear(in_dim, 256), nn.LayerNorm(256), nn.GELU(), nn.Dropout(p),
            nn.Linear(256, 64))
        self.ordinal  = CoralHead(64, K)    
        self.category = nn.Linear(64, 1)    
    def forward(self, x):
        z = self.proj(x)
        return z, self.ordinal(z), self.category(z).squeeze(-1)

ckpt = torch.load("KOZTAM.pt", map_location=DEVICE)
head = KOZTAMNet().to(DEVICE)
head.load_state_dict(ckpt["state_dict"])
head.eval()
thresholds = torch.tensor(ckpt["coral_thresholds"], device=DEVICE)   

CATEGORY = {0: "bilgilendirici (informative)", 1: "öyküleyici (narrative)"}

@torch.no_grad()
def predict(texts):
    single = isinstance(texts, str)
    cls   = encode_cls([texts] if single else list(texts))
    z     = head.proj(cls)
    score = head.ordinal.fc(z).squeeze(-1)                       
    grade = (score[:, None] > thresholds[None, :]).sum(1) + 1     
    cat   = (torch.sigmoid(head.category(z).squeeze(-1)) > 0.5).long()
    out = [{"grade": int(g), "score": round(float(s), 4), "category": CATEGORY[int(c)]}
           for g, s, c in zip(grade, score, cat)]
    return out[0] if single else out

print(predict("Sample text."))

Performance

Evaluated on the held-out test set (239 texts) the model never saw during training.

Headline metrics (calibrated).

Threshold calibration. KOZTAM has two parts: the trained network, which outputs a continuous difficulty score, and seven calibrated thresholds that cut that score into discrete grades (stored in the checkpoint, applied at inference). The raw network already ranks texts near-perfectly — continuous-score Spearman ρ = 0.966 — but the default 0.5-thresholds mis-placed the cut points and collapsed predictions toward the extreme grades. Re-placing the thresholds on the validation split (coordinate descent on MAE, no re-training) produced the final metrics:

On the validation split the same procedure moved MAE from 0.837 to 0.285, confirming the calibration generalizes rather than overfitting the test set.

Continuous difficulty score by grade (test). The mean score increases monotonically across all eight grades with no inversion, which is the direct evidence that the encoder learned the reading-difficulty ordering:

Per-grade MAE (calibrated, test).

The model is sharpest at the extremes (grades 1, 8) and at distinct mid-levels; its largest residual errors sit at the 3–4 and 6–7 boundaries, where texts are genuinely close in readability. On the raw predictions the two categories were balanced (informative 0.832 / narrative 0.842) and truncated texts (>512 tokens, n = 12) were only marginally harder (1.000 vs. 0.828), so neither category nor truncation is a source of systematic error.

Confusion matrix (calibrated, test). Rows = true grade, columns = predicted grade.

Ablation (test). Each loss component was removed in turn, with everything else — seed, initialization, batch order, scheduler, early stopping, and the same post-hoc threshold calibration — held identical.

Removing the ordinal-aware triplet loss raises MAE by 0.075 while pure ranking ρ is essentially unchanged, showing the triplet term improves the even spacing of the embedding axis (its calibratability) rather than the ranking itself.

Explainability (XAI)

Reading-difficulty attributions are produced with Integrated Gradients (IG) on an end-to-end version of the model (frozen BERTurk → continuous difficulty score), with token attributions merged back to whole words. Before attribution, the end-to-end score is verified against the cached score of the same text — they match to within 1.1 × 10⁻⁵, confirming IG explains the exact deployed model. IG is run with n_steps = 50, and the convergence delta is 0.056 (negligible relative to the score magnitude), so the completeness axiom holds.

Positive attribution means a word pushes the difficulty score up (harder); negative means it pushes down (easier). Three representative test cases are shown below. Because IG distributes each text's total score relative to an empty-content baseline, the absolute magnitude of the attributions scales with how far a text's score sits from that baseline — so magnitudes are comparable within a text, not across texts.

Case 1 — clear-easy (true grade 1, predicted 1).

Case 2 — clear-hard (true grade 7, predicted 7).

Case 3 — boundary error (true grade 6, predicted 7).

The boundary case is the most informative: the words that pushed a grade-6 text up into grade 7 are long, morphologically heavy Turkish words (alışkanlıklarımızda, yaşamımızdaki, cildimizin). This is direct evidence that the model reads morphological and lexical density as a difficulty signal, and that its single-grade error falls exactly on the kind of neighbouring boundary where the difficulty distinction is genuinely soft.