Devika: Malayalam Handwritten Word OCR

Devika is an open-source model for recognizing single handwritten Malayalam words from images. It is a fine-tuned version of Microsoft's trocr-large-handwritten with a new Malayalam character-level decoder vocabulary.

Scope: Devika is trained on word-level images. It reads one cropped word at a time. It is not a line, paragraph or full-page reader. See Limitations and Using Devika on full documents.

Versions

This repository holds two word-level models with the same architecture and the same task. They differ only in training data.

Both rows are measured on the same fixed 5,000-word random sample of the Bhashini test split, with no_repeat_ngram_size=0. See Results for details.

Which one should I use? On the Bhashini test words, the default model is clearly more accurate. The camera-captured model was additionally trained on camera-captured handwriting from many more writers, but that has not been evaluated yet, so this card does not claim it is more robust on phone-photographed text. If your input looks like camera-captured handwriting, evaluate both versions on a sample of your own data before choosing.

Load the default model:

model = VisionEncoderDecoderModel.from_pretrained("Artificial-Soul/devika_malayalam_trocr")
processor = TrOCRProcessor.from_pretrained("Artificial-Soul/devika_malayalam_trocr")

Load the camera-captured model:

model = VisionEncoderDecoderModel.from_pretrained("Artificial-Soul/devika_malayalam_trocr", revision="camera_captured")
processor = TrOCRProcessor.from_pretrained("Artificial-Soul/devika_malayalam_trocr", revision="camera_captured")

Results

Both versions were evaluated on the held-out Bhashini test split, which was never used for training or checkpoint selection. All numbers use beam search (5 beams, max length 32) with no_repeat_ngram_size=0.

Same 5,000-word random sample, both versions (fixed seed):

Word accuracy counts a word as correct only if every character matches, so a single wrong vowel sign or conjunct component fails the whole word. This is why it is much lower than character accuracy.

Default model checkpoint selection. The default model is the checkpoint at step 39,000 (about epoch 29), chosen by lowest validation CER. The validation CERs below were measured during training, before the decoding fix described in the Changelog, so they are not comparable to the numbers above:

Why the decoding setting matters. Decoding the default model with no_repeat_ngram_size=3 (a common default for subword tokenizers) gives 1.67% CER and 89.14% word accuracy on the full test split. That setting is wrong for a character-level tokenizer, because it forbids any word that repeats the same three characters. About 4% of test words do, and that setting got none of them right. With it turned off, the default model gets about 92% of those words right and the camera-captured model about 82%. The released configuration therefore sets no_repeat_ngram_size=0.

Model details

The encoder and decoder transformer layers are initialized from the English TrOCR checkpoint. The decoder's embedding and output layers were resized and re-learned for the Malayalam character vocabulary, since the original English BPE vocabulary cannot represent Malayalam.

Training data

The default model is trained on the Bhashini Mozhi Malayalam handwritten word-level dataset:

The training split comes with a vocabulary of 13,401 unique words. Ground truth is one transcription per cropped word image.

The camera-captured model is trained on the Mozhi data above plus word images from the Unconstrained Camera Captured Indic Offline Handwritten Dataset. That dataset consists of A4 pages written freehand by many writers (the dataset description mentions 1,220 writers across 13 Indic languages) and photographed with a mobile camera, with word-level annotations. The Malayalam word crops used here come with pre-built splits:

Both models are evaluated on the Mozhi test split only (see Results). Dataset references are listed under Citation.

Training procedure

• Method: full fine-tuning (all encoder and decoder weights updated), not LoRA
• Hardware: 4× NVIDIA A10G (24 GB), AWS EC2 g5.12xlarge, multi-GPU data parallel via 🤗 accelerate
• Per-GPU batch size: 16 (effective batch size 64)
• Learning rate: 3e-5, 10% warmup, linear decay
• Precision: fp16
• Label handling: an explicit end-of-sequence token is appended to every label so the model learns when to stop generating
• Checkpointing: evaluated every 1,000 steps on validation CER, and the best checkpoint by CER was kept
• Decoding used for the reported results: beam search with 5 beams, max length 32, no_repeat_ngram_size=0 (as set in each revision's generation_config.json)
• Camera-captured model: same architecture, tokenizer and training setup as the default model, trained on the combined dataset described above.

How to use

import torch
from PIL import Image
from transformers import TrOCRProcessor, VisionEncoderDecoderModel

repo_id = "Artificial-Soul/devika_malayalam_trocr"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = TrOCRProcessor.from_pretrained(repo_id)
model = VisionEncoderDecoderModel.from_pretrained(repo_id).to(device).eval()

image = Image.open("word.jpg").convert("RGB")   # a crop containing ONE word
pixel_values = processor(images=image, return_tensors="pt").pixel_values.to(device)

with torch.no_grad():
    generated_ids = model.generate(pixel_values, max_length=32, num_beams=5)

print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])

You may see a warning that encoder.pooler.* weights are newly initialized. This is harmless: TrOCR never uses that layer.

Limitations

• Single words only. The model was trained exclusively on isolated word crops. Feeding it a sentence, line or paragraph image produces poor output, because the image encoder squeezes the whole image to a fixed 384×384 input and the model has never seen multi-word images.
• Malayalam-only output alphabet. The vocabulary contains only Malayalam-block characters and a hyphen. The model cannot output spaces, English letters, Western digits (0–9) or other punctuation. Malayalam digits are supported, and chillu letters come out as single code points (ൻ, ർ, ൽ, ൾ, ൺ, ൿ). English words in mixed documents need a separate recognizer.
• Very wide word crops are compressed horizontally by the fixed-size resize, which can hurt accuracy on fine conjunct details.
• Evaluation covers the Bhashini test split only. Neither version has been measured on the camera-captured test words, or on other handwriting styles, low-quality photos, unusual pens or backgrounds. Accuracy on words very different from the training vocabulary (for example rare proper nouns) may also be lower. In particular, the camera-captured model saw camera-captured handwriting in training, but whether that helps on such input is untested.
• Not for unsupervised use on high-stakes documents. Even at roughly 98–99% character accuracy, errors occur. For legal, official or financial records, keep a human review step and use confidence-based flagging.

Using Devika on full documents

To read a full page, run a text detector first, split the page into lines, split each line into words, and pass each word crop to Devika. Join the words with spaces and the lines with newlines. Text detectors such as PaddleOCR or docTR can provide the line boxes.

Intended use

Malayalam handwritten word recognition for research, education, digitization pipelines (as the recognition stage after word segmentation), and as a base for further fine-tuning on domain-specific handwriting.

Licensing note

The model weights are released under the MIT license, matching the base TrOCR model. Please also respect the terms of the Bhashini dataset used for fine-tuning.

Acknowledgements

• Microsoft Research for TrOCR and the pretrained trocr-large-handwritten checkpoint. This model builds directly on their work.
• CVIT, IIIT Hyderabad (Minesh Mathew, Ajoy Mondal, C V Jawahar) for the Mozhi handwritten word dataset and the Unconstrained Camera Captured Indic Offline Handwritten Dataset.
• Anyone using these models: please check each dataset's license terms before redistributing derived work.
• Compute: training was run on AWS EC2 g5.12xlarge (4× NVIDIA A10G), provided by C-DIT.
• Hugging Face 🤗 transformers and accelerate for the training and model tooling.

Changelog

• Camera-captured model (camera_captured branch): added a second word-level model trained on Bhashini plus camera-captured handwriting.
• Decoding fix (both versions): no_repeat_ngram_size changed from 3 to 0 in generation_config.json. On a 5,000-word test sample this improved the default model from 1.72% to 1.11% CER (89.52% to 93.34% word accuracy) and the camera-captured model from 2.66% to 2.00% CER (84.92% to 88.30% word accuracy).

Citation

If you use these models, please cite the datasets they were trained on and the original TrOCR paper.

Mozhi dataset (both models): project page

@inproceedings{mathew2025towards,
  title        = {Towards Deployable OCR Models for Indic Languages},
  author       = {Mathew, Minesh and Mondal, Ajoy and Jawahar, CV},
  booktitle    = {International Conference on Pattern Recognition},
  pages        = {167--182},
  year         = {2025},
  organization = {Springer}
}

Unconstrained Camera Captured Indic Offline Handwritten Dataset (camera-captured model): project page

Ajoy Mondal and C V Jawahar, Unconstrained Camera Captured Indic Offline Handwritten Dataset, .

TrOCR (base model)

@inproceedings{li2023trocr,
  title     = {TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models},
  author    = {Li, Minghao and Lv, Tengchao and Chen, Jingye and Cui, Lei and Lu, Yijuan and Florencio, Dinei and Zhang, Cha and Li, Zhoujun and Wei, Furu},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year      = {2023},
  url       = {https://arxiv.org/abs/2109.10282}
}