trocr-base-rukopys-formula

A TrOCR-base (VisionEncoderDecoderModel, ~334M params) recognizer for handwritten mathematical expressions in Ukrainian documents, fine-tuned on the formula regions of the Rukopys dataset through a staged formula curriculum: HME100K → CROHME → synthetic Cyrillic formulas (noisy profile) → Rukopys silver → Rukopys gold.

TL;DR

Intended use

Recognizing cropped formula regions from handwritten Ukrainian documents (school-style math, chemistry, unit arithmetic), downstream of a layout detector. The output space is the normalized Rukopys formula convention.

For the plain-text region classes, see Hukyl/trocr-rukopys and Hukyl/trocr-large-rukopys.

How to use

from PIL import Image
from transformers import VisionEncoderDecoderModel, TrOCRProcessor

repo = "Hukyl/trocr-base-rukopys-formula"
processor = TrOCRProcessor.from_pretrained(repo)
model = VisionEncoderDecoderModel.from_pretrained(repo).eval()

crop = Image.open("formula_crop.png").convert("RGB")   # one detected formula region
pixel_values = processor(images=crop, return_tensors="pt").pixel_values
generated = model.generate(pixel_values, num_beams=4, max_length=256)
text = processor.batch_decode(generated, skip_special_tokens=True)[0]
print(text)

Output format

Rukopys formula labels are dual-mode: mostly plain linear math (R = 5, 7/10 y - 2/10 y = 5,5), with a minority of structural LaTeX (\frac, \int, \begin{array}, Ukrainian inside \text{…}). Training targets are normalized: LaTeX symbol commands become Unicode (\pi → π, \cdot → ·) while structural commands survive. The model emits that normalized space — Unicode symbols, slash fractions, Cyrillic words and units, and structural LaTeX where the source notation is genuinely two-dimensional.

Lineage & training curriculum

Staged curriculum; each stage fine-tunes the previous checkpoint. Optimizer: AdamW via the 🤗 Seq2SeqTrainer, linear LR decay, warmup ratio 0.1, max_target_length 256, seed 42. Every fine-tune stage uses the hybrid partial freeze: trainable are the upper half of the encoder blocks (plus the final encoder layernorm), the upper half of the decoder self-attentions, and all decoder cross-attentions; the lower blocks, embeddings, and LM head stay frozen. Intermediate stages select checkpoints by validation loss, the gold stage by validation CER.

1. Base — kazars24/trocr-base-handwritten-ru: microsoft/trocr-base-handwritten fine-tuned for Russian handwriting.
2. Rukopys line recognizer — Hukyl/trocr-rukopys: silver→gold fine-tune on the handwritten/printed/annotation/table crops (gold-val CER 0.1146).
3. HME100K (3 ep): handwritten mathematical expressions with LaTeX targets, normalized into the target space above.
4. CROHME (3 ep): the ICDAR/ICFHR competition corpus; InkML stroke traces rendered to 384px raster images, labels normalized like HME100K.
5. Synthetic Cyrillic formulas, noisy profile (3 ep): an in-house generator font-renders template-sampled formula strings — fractions, equations, square roots, angles, unit arithmetic, sequences, chemistry, mixed fractions, multi-step expressions, matrices — with Cyrillic variables and Ukrainian unit words (см, кг, год, …), and render-time degradation (rotation ±2°, Gaussian blur, additive noise, baseline jitter, random fonts and sizes). The noisy profile weights the mix toward the harder families (chemistry, mixed fractions, multi-step, matrices, deliberately ambiguous short forms) that the balanced profile leaves out.
6. Rukopys silver formula pretrain (5 ep): auto-labeled formula crops from the Rukopys silver split.
7. Rukopys gold formula fine-tune (15 ep): the human-labeled formula crops — the published weights.

Hyperparameters (gold stage, as launched)

The full recorded recipe ships in this repo's training_meta.json.

Results

Protocol: greedy decode, the formula validation split of the Rukopys gold train data (seed 42, crop-level split), full split. Labels are metric-normalized as described in Output format.

Compared runs

Same curriculum and gold-stage recipe unless noted; all compared checkpoints are internal and unpublished.

This model was evaluated on the full validation split; the compared runs on a 2,000-sample subsample. The gap to the balanced-profile run is within evaluation noise.

Limitations & biases

• Outputs are normalized formula text, not faithful LaTeX transcription — not a general image-to-LaTeX model.
• Handwritten Ukrainian school-style material only; expect degradation on other scripts, printed math, or clean typeset input.
• The train/val split is stratified at crop level, not page level — crops from the same page (and writer) appear on both sides, so the reported val CER is optimistic compared to a page-level protocol.
• Single seed and validation split — no across-run variance estimate.

Training data & attribution

The base model kazars24/trocr-base-handwritten-ru declares no explicit license — verify upstream terms before redistribution.