trocr-rukopys

A TrOCR-base (VisionEncoderDecoderModel, ~334M params) line recognizer for handwritten Ukrainian document text, fine-tuned on the Rukopys dataset.

The training data included handwritten, printed, annotation and table region crops from Rukopys.

TL;DR

Intended use

Recognizing cropped line/region images from handwritten Ukrainian documents, downstream of a layout detector.

For a stronger recognizer from a later, larger training generation, see Hukyl/trocr-large-rukopys.

How to use

from PIL import Image
from transformers import VisionEncoderDecoderModel, TrOCRProcessor

repo = "Hukyl/trocr-rukopys"
processor = TrOCRProcessor.from_pretrained(repo)
model = VisionEncoderDecoderModel.from_pretrained(repo).eval()

crop = Image.open("region_crop.png").convert("RGB")          # one detected line/region
pixel_values = processor(images=crop, return_tensors="pt").pixel_values
generated = model.generate(pixel_values, max_new_tokens=256)  # greedy
text = processor.batch_decode(generated, skip_special_tokens=True)[0]
print(text)

Lineage & training curriculum

Two-stage silver→gold fine-tuning from a Russian handwriting base. Optimizer: AdamW via the 🤗 Seq2SeqTrainer, linear LR decay, max_target_length 256, online augmentation, seed 42.

1. Base — kazars24/trocr-base-handwritten-ru: microsoft/trocr-base-handwritten fine-tuned for Russian handwriting.
2. Silver pretrain (5 ep): partial-freeze pretrain on the Rukopys silver split (~119.8k auto-labeled crops).
3. Gold fine-tune (up to 15 ep, early stopping with patience 3): full encoder + decoder unfreeze on the human-labeled gold train split, 19,069 train / 3,366 val crops, stratified at crop level.

Hyperparameters (as launched)

The full recorded recipe ships in this repo's training_meta.json.

Online data augmentation

Online augmentation was applied during training. Only training crops were augmented (no augmented validation was measured).

Each crop was transformed once per epoch by one geometric + one or two photometric operations at random. Each class received a separate augmentation profile that was selected to minimize the distribution shift.

These simulate scanner/paper variation, ink thinning/bleed, and natural handwriting deformation, widening the range of appearances beyond the raw training crops.

Results

Protocol: greedy decode, max_new_tokens=512, 1000-sample draw from the gold validation split, seed 42. Labels are metric-normalized (LaTeX → Unicode, table PSV canonicalized) so CER/WER reflect the scored character set.

Overall

Per class

We also acknowledge that printed/table/annotation n is quite small, so measuring CER against them is quite noisy.

Limitations & biases

• Handwritten Ukrainian archival document material only; expect degradation on other scripts, languages, or modern born-digital text.
• The character set and normalization are tuned to the document-OCR metric — outputs are normalized text, not faithful transcription.
• The train/val split is stratified at crop level, not page level — crops from the same page (and writer) appear on both sides, so the reported val CER is optimistic compared to a page-level protocol.
• annotation and table classes are weak.
• Single seed and validation split — no across-run variance estimate.

Training data & attribution

The base model kazars24/trocr-base-handwritten-ru declares no explicit license — verify upstream terms before redistribution.