Telugu OCR — line recognition, trained from scratch

A Telugu text-line recogniser built end to end for Telugu: a convolutional-stem CTC image encoder joined to a Telugu grapheme language model by cross-attention. Give it an image of one line of Telugu text and it returns the text.

It makes about one fifth as many errors as the best general-purpose OCR engine available for Telugu.

Results

Scored on harsha-desaraju/telugu-line-ocr-bench — 1044 line crops whose transcriptions a human checked against the image. Every engine read the identical images and was scored against the identical references. All figures are percentages; lower is better except the last column.

AER — akshara error rate — is the number to read for Telugu. A Telugu akshara is usually three or four code points, so CER charges a missing vowel sign one edit and a garbled cluster two or three, while a reader perceives one wrong syllable either way.

By AER this model is 5.0× better than Tesseract, 6.8× better than Surya and 7.0× better than PaddleOCR. It transcribes 71.7% of lines with zero errors, against 26.7% for the strongest baseline — the difference between output you can read and output you have to correct.

Its remaining errors: 244 substitutions, 78 insertions, 205 deletions across 26,650 reference aksharas. Deletions dominate, mostly dropped spaces.

Usage

The architecture is custom, so the model ships its own code and needs trust_remote_code=True. Nothing else to install beyond transformers, torch, pillow and regex.

from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("harsha-desaraju/telugu-ocr-model",
                                  trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained("harsha-desaraju/telugu-ocr-model",
                                          trust_remote_code=True)

# one image -> str
print(model.transcribe(Image.open("line.png"), tokenizer))

# a list of images -> list[str]
print(model.transcribe([Image.open(p) for p in paths], tokenizer))

# fast mode: no beam search, ~9x quicker for ~11% more error
print(model.transcribe(images, tokenizer, decode="ctc"))

On a GPU, move the model first: model = model.to("cuda").

Input requirements

Feed it one line of text per image. Preprocessing is handled internally — grayscale, resize to height 64 preserving aspect ratio, pad the width to a multiple of 8 — so pass a plain PIL.Image. A line wider than 2048px once scaled to height 64 cannot be encoded and returns "" rather than being silently truncated.

There is no page layout analysis or line detection here. Segment lines yourself first.

Decode modes

The same weights support three decoding strategies, selected per call.

joint is the headline number and the default. Reach for ctc when labelling millions of crops: it costs ~11% more AER and runs about nine times faster, with no beam search at all.

Architecture

~81M parameters, float32.

line image (1×64×W, W ≤ 2048)
   ↓  conv stem: 6 blocks, 32→64→128→256→320→384, height→1, width÷8
   ↓  10-layer pre-LN transformer encoder, d=384, 8 heads, MLP 1536
   ├──→ CTC head (2049 classes = 2048 aksharas + blank)
   ↓  enc_to_dec: linear 384 → 512
   ↓  16-layer GPT decoder, d=512, SwiGLU 1368, 8 heads, ctx 256
   ↓  cross-attention on every second block, zero-init tanh gate
   →  text

Counts are over the checkpoint's 388 tensors, so they include the decoder's non-trainable positional table.

Two design choices carry most of the quality:

Grapheme (akshara) tokenisation. Text is split with regex.\X into Unicode grapheme clusters — one cluster, one token, no BPE. Vocabulary 2048. A Telugu syllable is a single token, so the decoder predicts syllables rather than assembling them from fragments of code points.

Two heads over one encoder. The CTC head sees the image directly and cannot hallucinate text that is not there; the attention decoder carries language knowledge and repairs what CTC garbles. Keeping both allows rescoring one against the other, which is where the best numbers come from.

Training

Three pretrained pieces, then two fine-tuning stages:

1. Image encoder — trained with CTC loss on Telugu line images.
2. Text decoder — a 16-layer causal grapheme language model pretrained on Telugu text.
3. Stage 1 — both backbones frozen; only the new cross-attention, its norms, the gates and the bridge are trained. The gate is zero-initialised behind a tanh, so the image contributes nothing at initialisation and is phased in as the gate learns.
4. Stage 2 — everything unfrozen and trained end to end, with the encoder's CTC head kept as an auxiliary term so it cannot drift away from the features the decoder relies on.

Training data was synthetic Telugu line renders (many fonts and sizes, degraded with a scan-simulation pipeline) mixed with real line crops from Telugu Wikisource proofread page scans. Real crops were not augmented — they are already degraded.

Limitations

• Printed book text only. Trained on printed Telugu from book scans and synthetic renders. No handwriting, no signage, no scene text, no historical orthography.
• Single lines only. No layout analysis or line detection; segment before calling.
• Hard width ceiling of 2048px at height 64. Longer lines must be split.

Citation

@misc{telugu_ocr_model,
  title  = {Telugu OCR: a grapheme-level CTC encoder with a cross-attending Telugu language model},
  author = {Desaraju, Harsha},
  year   = {2026},
  url    = {https://huggingface.co/harsha-desaraju/telugu-ocr-model}
}

Code: github.com/harsha-desaraju/TeluguOCR