sinhala-print-trocr-pilot
Status: frozen pilot. Superseded by kaan84/sinhala-print-trocr.
This repository is kept unchanged as the published record of a tokenizer defect described below. It contains a known bug and should not be used for new work. Use the successor repo instead.
| this repo | successor | |
|---|---|---|
| name | sinhala-print-trocr-pilot | sinhala-print-trocr |
| role | frozen "before" state | current model |
| tokenizer | deletes Zero Width Joiner | ZWJ preserved |
| use it? | no — reference only | yes |
What this model is
A TrOCR fine-tune for printed Sinhala text lines, trained on synthetic data rendered from five fonts. Input is a single line image (~2600×128); it is not a page or document recogniser.
Known defect: the decoder tokenizer deletes ZWJ
The decoder tokenizer's BertNormalizer runs with clean_text=True, which strips
every Unicode Cf (format) character — including U+200D ZERO WIDTH JOINER. In
Sinhala, ZWJ is what binds consonants into conjuncts such as ශ්රී and ප්රජා.
tok = AutoTokenizer.from_pretrained("kaan84/sinhala-print-trocr-pilot")
s = "ශ්\u200Dරී"
tok.decode(tok.encode(s, add_special_tokens=False)) == s # False
The character is removed silently: no error, no warning, and no signal in the loss curve. The model therefore cannot emit a ZWJ conjunct at any point in training.
How much text this affects
Measured on the training corpus (6,968 usable lines after filtering one null row):
| unit | affected | share |
|---|---|---|
| codepoints | 414 / 487,209 | 0.085% |
| grapheme clusters | 407 / 344,999 | 0.118% |
| lines (≥1 conjunct) | 315 / 6,968 | 4.52% |
An earlier description of this work cited ~6.6% of training text. That figure was wrong. The corrected measurements are above; the line-level figure (4.52%) is the closest to the original claim, and it is the weaker statement — a line counts as affected if it contains a single affected character.
Grapheme clusters are computed with
grapheme-kit, which folds ZWJ sequences
into a single cluster. Default UAX #29 segmentation does not do this and will
undercount.
Reported results
Evaluated on a 5% held-out split (349 lines, seed 42) of the training dataset.
| metric | value |
|---|---|
| char-CER (codepoint) | 0.1820 |
| g-CER (grapheme cluster) | 0.1852 → see successor |
| conjunct recall | 0.0000 (0/19) |
Conjunct recall is 0 by construction, not by difficulty.
These numbers do not measure reading ability. Training and evaluation come from
the same synthetic generator and the same five fonts, so they largely measure
memorisation of one renderer. Real scanned documents are substantially harder — a
single page from
avishadilhara/sinhala-ocr-lk-acts-1010
scored char-CER 0.4657 with line segmentation applied.
Intended use
Reference and reproduction only. For anything else use
kaan84/sinhala-print-trocr.
Citation of the underlying issue
The general problem of pre-tokenization and grapheme handling in complex scripts is treated in:
- Velayuthan & Sarveswaran (2025), Egalitarian Language Representation in Language Models: It All Begins with Tokenizers, COLING 2025.
- Nisfer et al. (2026), grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP.
Note that the defect in this repository is in the normalizer, not the
pre-tokenizer: BertPreTokenizer preserves ZWJ intact and yields one pre-token per
conjunct.