🟢 CANINE Persian Diacritizer (canine-fa-diacritizer)

Restore diacritics (حَرَکات / اِعراب) on undiacritized Persian text — one short-vowel prediction per character.

Persian is almost always written without short vowels, which makes the same letter string ambiguous to read aloud (مَرد "man" vs مُرد "died"). This model reads a raw Persian sentence and puts the missing marks back — Fatha, Damma, Kasra, Sokun, Shadda, Fathatan and the Shadda+vowel combinations — so the text becomes fully vocalized.

It's built on Google's CANINE, a tokenizer-free encoder that reads one Unicode character at a time. That's the whole trick: a normal subword tokenizer would glue several letters into one opaque token, so you couldn't attach a diacritic to each individual letter. CANINE gives a clean 1-label-per-character alignment, which is exactly what diacritization needs.

• Task: Persian diacritization, framed as character-level token classification (10 classes)
• Base model: google/canine-s (Apache-2.0)
• Parameters: ~132M
• Language: Persian / Farsi (fa)
• License: MIT

🚀 How to use

This is the important part. The model outputs one label per input character; you then re-insert the corresponding diacritic mark after each character. The snippet below is fully self-contained — no extra files, no local modules.

import torch
from transformers import CanineTokenizer, CanineForTokenClassification

MODEL_ID = "PedramR/canine-fa-diacritizer"

tokenizer = CanineTokenizer.from_pretrained(MODEL_ID)
model = CanineForTokenClassification.from_pretrained(MODEL_ID).eval()

# Combining marks, keyed by the label-name pieces stored in model.config.id2label.
MARKS = {
    "FATHA": "َ", "DAMMA": "ُ", "KASRA": "ِ",
    "SOKUN": "ْ", "SHADDA": "ّ", "FATHATAN": "ً",
}

def label_to_marks(label: str) -> str:
    # e.g. "SHADDA_KASRA" -> ّ + ِ ; "NONE" -> ""
    if label == "NONE":
        return ""
    return "".join(MARKS[part] for part in label.split("_"))

@torch.no_grad()
def diacritize(text: str) -> str:
    enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=2048)
    logits = model(**enc).logits
    preds = logits.argmax(-1)[0].tolist()

    # CANINE adds [CLS] at index 0 and [SEP] at the end; everything in between
    # aligns 1:1 with the input characters.
    char_preds = preds[1:1 + len(text)]

    id2label = model.config.id2label
    out = []
    for ch, pid in zip(text, char_preds):
        out.append(ch)
        out.append(label_to_marks(id2label[pid]))
    return "".join(out)


print(diacritize("کتاب من"))
# -> کِتاب مَن

Why the preds[1:1 + len(text)] slice? CANINE is character-level and adds a [CLS] token at the start and [SEP] at the end. Dropping the first prediction and taking exactly len(text) after it re-aligns predictions perfectly with your original characters. Do not strip or normalize the input first — feed the raw string so the alignment holds.

🏷️ Label scheme (10 classes)

The model predicts one of these per character. They are stored in model.config.id2label, so you never need to hard-code them.

Shadda (gemination) stacks with a vowel on the same base letter, so the combinations get their own compound labels rather than being dropped.

📊 Evaluation

Evaluated on a held-out test set of 359 sentences (59,927 characters, 17,273 words). Metrics follow the Arabic-diacritization convention (Fadel et al., 2019 / CATT). The starred variants (DER* / WER*) exclude each word's final character, which in Persian is usually the syntax-dependent Ezafe Kasra and is genuinely harder than the rest of the word.

🧠 Training

• Data: avaeziaiteam/harakat-dataset (private) — pairs of raw and diacritized Persian sentences. Base characters and per-character labels are derived by stripping the marks out of the diacritized text itself, which guarantees exact 1:1 char↔label alignment.
• Splits: 7,180 deduplicated sentences → 6,462 train / 359 dev / 359 test (90 / 5 / 5, seed 42).
• Objective: token classification, best checkpoint chosen by macro-F1 over diacritic classes.
• Hyperparameters: 8 epochs · lr 5e-5 (cosine, 10 % warmup) · batch size 16 · weight decay 0.01 · AdamW · fp16 · length-grouped batching · truncation at 512 characters.
• Hardware: single NVIDIA RTX 3090.
• Frameworks: transformers, torch, datasets.

⚠️ Intended use & limitations

Intended use. Restoring diacritics on modern written Persian — as a front-end for text-to-speech, a reading aid for learners, or a preprocessing step for linguistic tooling.

Limitations.

• Trained on a modest (~6.5K-sentence) corpus; expect weaker results on domains far from the training data (e.g. classical poetry, heavy technical jargon, dialectal text).
• Word-final position is the hard part. The Ezafe Kasra at the end of a word depends on syntax that can reach beyond the sentence, so errors concentrate there — hence the separately reported DER*/WER*. SHADDA_KASRA is the weakest class (F1 0.806).
• Training truncated inputs to 512 characters; at inference the tokenizer allows up to 2048 (CANINE's architectural max is 2046). For long documents, split into sentences.
• The rare SUPERSCRIPT_ALEF mark (9 occurrences in the whole corpus) is folded into NONE and is not predicted.

📄 License & attribution

• Model & code: MIT © 2026 Pedram Rostami.
• Base model: google/canine-s — Apache-2.0.
• Metric definitions follow Fadel et al. (2019) and the CATT Arabic-diacritization line of work.