ByT5 Persian ↔ Tajik Transliterator
Bidirectional transliteration between Persian (Perso-Arabic script) and Tajik (Cyrillic
script), fine-tuned from google/byt5-small on the
ParsText Persian–Tajik parallel corpus. ByT5 operates on raw UTF-8 bytes, so both scripts are
handled natively without script-specific tokenization.
Direction is selected with a task prefix:
fa2tg: …— Persian → Tajik (Cyrillic)tg2fa: …— Tajik → Persian (Perso-Arabic)
| Architecture | ByT5-small (byte-level encoder–decoder) |
| Parameters | 299 M |
| Base model | google/byt5-small |
| Training data | ParsText Persian–Tajik parallel corpus |
| License | MIT |
Intended Use
- Persian ↔ Tajik script conversion for cross-script search and reading
- Script normalization ahead of Persian/Tajik NLP pipelines that assume a single script
- A starting point for fine-tuning on related low-resource script-conversion tasks
Usage
Shekar
Shekar wraps 8-bit quantized ONNX exports of the encoder and decoder, so it runs on CPU without
PyTorch. FarsiToTajik and TajikToFarsi set the direction prefix for you.
from shekar import FarsiToTajik, TajikToFarsi
to_tajik = FarsiToTajik()
to_farsi = TajikToFarsi()
print(to_tajik("ایران مادر است!")) # Эрон модар аст!
print(to_farsi("Донишгоҳи Теҳрон")) # دانشگاه تهران
Both accept num_beams (default 1) and max_new_tokens (default 256).
Transformers
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
repo = "shekar-ai/byt5-small-tajik-farsi-translit"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo).eval()
@torch.no_grad()
def transliterate(text, direction="fa2tg", num_beams=4):
assert direction in {"fa2tg", "tg2fa"}
inputs = tokenizer(f"{direction}: {text.strip()}", return_tensors="pt", truncation=True, max_length=1024)
output = model.generate(**inputs, num_beams=num_beams, max_new_tokens=1024, early_stopping=True)
return tokenizer.decode(output[0], skip_special_tokens=True).strip()
print(transliterate("دانشگاه تهران", "fa2tg")) # Донишгоҳи Теҳрон
print(transliterate("Эрон модар аст!", "tg2fa")) # ایران مادر است!
Training
Data. The ParsText Persian–Tajik parallel dataset. After cleaning (non-empty pairs, length ≤ 200
characters) and bidirectional expansion (one example per direction), the corpus contains 751,650
examples, split 90/5/5 into train/validation/test. Persian text was normalized with Shekar's
preprocessing pipeline (AlphabetNormalizer → YaNormalizer → SpacingNormalizer).
| Hyperparameter | Value |
|---|---|
| Epochs | 5 (early-stopped on best chrF++) |
| Batch size | 32 per device |
| Learning rate | 5e-4 |
| LR scheduler | Linear, 5% warmup |
| Weight decay | 0.01 |
| Precision | bf16 mixed |
| Beam search (eval) | 4 beams |
| Max sequence length | 1024 bytes |
Evaluation
Held-out test set of 37,582 examples, scored with chrF++ (character n-gram F-score,
word_order=2), CER (character error rate), and exact sequence match:
| Direction | n | chrF++ ↑ | CER ↓ | Sequence accuracy ↑ |
|---|---|---|---|---|
| Overall | 37,582 | 89.68 | 0.0382 | 0.537 |
fa2tg (Persian → Tajik) | 18,705 | 87.90 | 0.0467 | 0.423 |
tg2fa (Tajik → Persian) | 18,877 | 91.76 | 0.0283 | 0.649 |
tg2fa scores higher because the Perso-Arabic script collapses several Tajik vowel distinctions,
making the target more deterministic than in the reverse direction.
Limitations
Persian → Tajik is the harder direction: Persian orthography leaves short vowels unwritten, so the model must infer them, and exact-match accuracy is correspondingly lower (42% vs 65%). Training examples were capped at 200 characters — transliterate long text sentence by sentence. Proper nouns and loanwords absent from ParsText are the most common source of errors.
Citation
@article{Amirivojdan2025Shekar,
author = {Amirivojdan, Ahmad},
title = {{Shekar: A Python Toolkit for Persian Natural Language Processing}},
journal = {Journal of Open Source Software},
volume = {10},
number = {114},
pages = {9128},
year = {2025},
doi = {10.21105/joss.09128},
url = {https://joss.theoj.org/papers/10.21105/joss.09128}
}