HATFormer Fine-Tuned (AraMS-28k)

Line-level handwritten text recognition for historical Arabic manuscripts. A HATFormer-style VisionEncoderDecoder (ViT-384 encoder + TrOCR decoder, 334 M parameters) fine-tuned on line-level annotations from AraMS-28k-HTR.

This model needs custom preprocessing. Lines are resized to height 64, flipped horizontally (Arabic is RTL), then pasted into a 384×384 canvas — wide lines stacked as rows. Feeding a plain TrOCRProcessor image produces nonsense, which is why the inference widget is disabled. See Usage.

Results

Character Error Rate, by how far the test data sits from training:

The last row is the character-weighted micro-average over the three held-out manuscripts — the honest generalization number. The spread across them is the real story: script tradition dominates. Maghrebi, whose letterforms and diacritic conventions differ most from the training mix, costs ~25 CER points over Ruqʿah.

CER is micro-averaged (total edits ÷ total reference characters), with the same Arabic normalization applied to predictions and references so that models trained under different diacritization conventions are compared fairly.

The in-distribution row comes from a held-out-pages split of the training manuscripts. The published dataset is split at manuscript level (train/val/ test), so that particular condition is not reproducible from it directly.

Usage

import numpy as np, torch
from huggingface_hub import snapshot_download
from PIL import Image
from transformers import PreTrainedTokenizerFast, VisionEncoderDecoderModel

# Download first, then load from disk — see the note below on transformers versions.
CKPT = snapshot_download("Archatext/hatformer-arams28k")
model = VisionEncoderDecoderModel.from_pretrained(CKPT).eval()
tok = PreTrainedTokenizerFast.from_pretrained(CKPT)
tok.add_special_tokens({"pad_token": "<pad>", "eos_token": "</s>",
                        "cls_token": "<s>", "bos_token": "<s>"})
model.config.decoder_start_token_id = tok.bos_token_id
model.config.pad_token_id = tok.pad_token_id


def hatformer_canvas(img, height=64, canvas=384):
    """Line image -> the 384x384 RTL-flipped, row-stacked canvas the model expects."""
    img = img.convert("RGB")
    w, h = img.size
    resized = img.resize((max(1, int(height * w / h)), height)).transpose(Image.FLIP_LEFT_RIGHT)
    out = Image.new("RGB", (canvas, canvas), (0, 0, 0))
    if resized.width <= canvas:
        out.paste(resized, (0, 0))
    else:                                    # wide lines wrap into stacked rows
        for i in range((resized.width + canvas - 1) // canvas):
            seg = resized.crop((i * canvas, 0, min((i + 1) * canvas, resized.width), height))
            out.paste(seg, (0, i * height))
    return out


def read(path, num_beams=3):
    arr = np.asarray(hatformer_canvas(Image.open(path)), dtype=np.float32) / 255.0
    arr = (arr - 0.5) / 0.5                                   # ViT mean/std, applied directly
    pv = torch.from_numpy(np.transpose(arr, (2, 0, 1)))[None]
    ids = model.generate(pixel_values=pv, num_beams=num_beams, length_penalty=0, max_new_tokens=64)
    return tok.batch_decode(ids, skip_special_tokens=True)[0]


print(read("line.png"))

The canvas is already exactly 384 px, so ViT normalization (rescale 1/255, then mean/std 0.5) is applied directly — no TrOCRProcessor needed. Beam width 3 and length_penalty=0 match the evaluation recipe.

Why snapshot_download first? This repo ships weights as model.safetensors only. With transformers==4.37.2 — the version these numbers were produced with, pinned in the code repo for its VisionEncoderDecoder generation path — passing the repo id straight to from_pretrained fails with "make sure … a file named pytorch_model.bin": that release cannot resolve a safetensors-only repo through current huggingface_hub. Downloading the snapshot and loading from the local path works on every version, newer transformers included.

Batch evaluation with per-manuscript breakdowns: scripts/eval_hatformer.py in the code repo.

Model details

Training data and provenance

Fine-tuned on the train split only of AraMS-28k-HTR — 20,103 lines from 9 manuscripts. The val split (2 manuscripts) was used for model selection; the test manuscripts (book_03, book_05, book_09) were never trained on.

The initialization chain is worth stating plainly:

microsoft/trocr-base-stage1  ->  HATFormer fine-tuned on Muharaf  ->  this model

This is not a Muharaf-free initialization. It matters when comparing against models trained from a different starting point: some of the Arabic-script competence here predates the AraMS-28k-HTR fine-tune.

Intended use

Line-level HTR on historical Arabic manuscripts, and as the frozen OCR critic in AraMS-Restore, where its recognition loss trains a restoration U-Net toward legibility. That second role is why the training split matters: the critic must not have seen the manuscripts the restoration models are tested on, or both the training signal and the metric are contaminated.

Limitations

• Script-dependent. CER ranges from 13.26 (Ruqʿah) to 37.88 (Maghrebi) across unseen manuscripts. Treat it as usable for in-distribution scripts and as a measuring instrument elsewhere, not as a general Arabic HTR model.
• Line-level only. Page segmentation is upstream; the model does not localize lines.
• Requires the custom preprocessing above. The RTL flip in particular is easy to miss and silently destroys accuracy.
• Trained on 14 manuscripts from one collection — no claim of generalization to other hands, papers, or periods.
• Non-commercial license, inherited from the source material.

Citation