Naqta - Arabic Punctuation Restoration
Restores punctuation in Modern Standard Arabic. Two components in one repo:
| component | path | task |
|---|---|---|
| point-mark model | repo root | 8 classes (XLM-RoBERTa-large) |
| span tagger | span-tagger/ | paired interjection dashes (BIO, XLM-RoBERTa-base) |
Usage
from transformers import AutoModelForTokenClassification, AutoTokenizer
tok = AutoTokenizer.from_pretrained("MostafaMaroof/Naqta")
model = AutoModelForTokenClassification.from_pretrained("MostafaMaroof/Naqta")
span_tok = AutoTokenizer.from_pretrained("MostafaMaroof/Naqta", subfolder="span-tagger")
span_model = AutoModelForTokenClassification.from_pretrained(
"MostafaMaroof/Naqta", subfolder="span-tagger")
Use sliding-window inference for long text. The model was fine-tuned on short
windows; a single truncated pass leaves everything past roughly 230 words
unpunctuated. See app.py in the Space for a reference implementation.
Evaluation
Two evaluations on deliberately different data. The gap between them is the main practical finding: Arabic punctuation is convention-dependent, so in-domain scores overstate performance on text punctuated to an external editorial standard.
1. In-domain test set
Held-out split of the training corpus (books, Wikipedia, diacritised text) — same sources and editorial conventions as training.
| metric | score |
|---|---|
| Macro F1 | 0.8960 |
| Token accuracy | 0.9714 |
2. External guideline benchmark
91 Modern Standard Arabic passages (about 27k tokens) annotated by an independent Arabic language centre in Saudi Arabia, following its own published punctuation guideline. No overlap with the training data.
| metric | previous release | current |
|---|---|---|
| Macro F1 | 0.5464 | 0.6160 |
| Weighted F1 | 0.8990 | 0.9154 |
| Token accuracy | 0.9096 | 0.9133 |
Per-class F1 (current):
| mark | O | : | ؟ | . | ، | ؛ | - | ! |
|---|---|---|---|---|---|---|---|---|
| F1 | 0.967 | 0.821 | 0.812 | 0.700 | 0.647 | 0.454 | 0.273 | 0.255 |
The drop from 0.896 to 0.616 is concentrated in marks whose placement depends on editorial convention rather than syntax. Structurally determined marks (question mark, colon) transfer well; the semicolon and exclamation, whose rules differ between style guides, do not.
Limitations
- Exclamation and semicolon are over-predicted (precision 0.204 and 0.362): the model finds these marks but does not withhold them reliably.
- Brackets and quotation marks are not produced. The span tagger covers them in training but their precision is too low to enable in production.
- Expect lower accuracy on text following a punctuation standard different from the training sources.
Training
XLM-RoBERTa-large fine-tuned on about 1.3M paragraphs (classical and modern books, Wikipedia, news, QA corpora) with weighted focal loss, oversampling of rare marks, and a final adaptation stage on guideline-conforming data.