nara-qwen-3.5-0.8b
Qwen3.5-0.8B fine-tuned for cultural heritage transcription of English-language
archival documents — United States National Archives Revolutionary War pension
files: printed forms completed by hand, catalogue cards, and pages of 18th- and
19th-century American cursive.
Given a page image the model returns one line of text per physical written or printed line, reading the printed text and the handwriting together in the order they appear.
This is the finished 40,000-row model — all 3 epochs, 14,535 steps.
Provenance — a medieval Latin model retrained on English archives
Qwen/Qwen3.5-0.8B— the stock vision-language model.comma-qwen-3.5-0.8b-full-33k— 33,112 pages of medieval Latin under the CATMuS graphemic standard (cer_nfd0.1240).- NARA, 10,000 rows — 8,756 pages of this dataset, 3 epochs.
- NARA, 40,000 rows — 38,756 pages, 3 epochs. This model.
The medieval ancestry is not incidental. Before NARA training the model applied
CATMuS conventions to English: asked to read a pension form it wrote SERUICE
for SERVICE, because rule B5 of its previous prompt maps v to u. Undoing
that is a large part of what this fine-tuning did — and it is why this model
ships its own prompt.txt rather than the CATMuS one.
Results
The full held-out split: 762 pages from 39 documents that appear nowhere in training (0 documents shared with the 1,407 training documents). A document is held out whole, because the corpus averages ~27 pages per document in the same hand on the same printed form, so splitting by page would measure memorisation rather than reading.
| metric | this model | parent (10k NARA model) |
|---|---|---|
| CER (micro) | 0.2868 | 0.5718 |
| CER (macro) | 0.3822 | 0.8485 |
| WER | 0.3379 | 0.5921 |
| line recall | 0.1901 | 0.1653 |
| CER excl. degenerate pages | 0.2328 | 0.4256 |
| truncated pages | 1.3% | 3.9% |
| degenerate pages | 0.8% | 2.0% |
Against its parent: micro CER improved by 0.2850; macro CER improved by 0.4663; line recall improved by 0.0248.
Both columns were produced in the same job, on the same pages, by the same code, so they are an exact comparison rather than numbers from different runs. Greedy decoding, 3072 max new tokens, 2048 visual tokens per page, batch 4.
cer_nfd equals cer here to every digit: NFD normalisation is a no-op on
English ASCII. It matters for this project's medieval Latin models and not at
all for this one.
Read this as one collection, not as general English HTR. Every page — training and holdout alike — comes from US Revolutionary War pension files. The holdout is unseen documents, which is a real test of generalisation across hands and forms, but it is not a test of generalisation across archives.
The failure mode worth knowing about
Error on this material is not evenly spread, and the gap between the micro and macro figures above is the measurement of that. A typical page of connected prose scores far better than the headline; the error concentrates on sparse catalogue cards, where a model that has not learned to stop transcribes the printed grid rules until it hits the token ceiling. One such page — 43 reference characters against 6,141 generated — can contribute as much edit distance as a hundred ordinary pages, because micro CER is a ratio of totals.
Micro CER weights a page by its length, so it describes a long page of prose. Macro CER weights every page equally, so it describes the sparse cards. Quote the one that matches your material, and expect short, sparse pages to be considerably worse than the headline.
Two things reduce it: more training data (1,000 → 10,000 pages cut truncation
from 15% to 2% on the 2B, and 10,000 → 40,000 cut it further to 1.3%),
and repetition_penalty=1.1 at inference. Use the penalty when serving.
Usage
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
REPO = "wjbmattingly/nara-qwen-3.5-0.8b"
model = AutoModelForImageTextToText.from_pretrained(REPO, dtype=torch.bfloat16,
device_map="auto")
processor = AutoProcessor.from_pretrained(REPO)
# The prompt is part of the model. It ships in this repo as prompt.txt and must
# be used verbatim: the model was trained to answer this exact instruction, and
# a paraphrase is a different task than the one it was fine-tuned on. In
# particular, do NOT use the CATMuS prompt from this project's medieval models.
from huggingface_hub import hf_hub_download
prompt = open(hf_hub_download(REPO, "prompt.txt")).read()
image = Image.open("page.jpg").convert("RGB")
messages = [{"role": "user", "content": [
{"type": "image"}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True,
enable_thinking=False)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=3072, do_sample=False,
repetition_penalty=1.1)
print(processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:],
skip_special_tokens=True))
enable_thinking=False is required, not optional. Qwen3.5's chat template leaves
the thinking block open by default at 4B and above; a model trained on a
closed block will then reason instead of transcribing and never reach the text.
The repo's processor_config.json is already capped to 2,048 visual tokens,
the budget training and scoring used. Loading a stock processor instead shows the
page at the library default of ~16k tokens — a resolution the model never saw.
Training
Full fine-tune (852,985,920 parameters, all of them), 3 epochs, 14,535 steps, learning rate 5e-5, effective batch 8, bf16, on one RTX PRO 6000 Blackwell (34 h).
Training data
wjbmattingly/si-test —
38,756 pages from 1,407 documents, the first 40,000 rows of the
dataset. Targets are line-by-line transcriptions; --- marks a horizontal rule.
Intended use and limits
Built for cultural heritage and digital humanities work: making handwritten archival material searchable and readable at scale. It is a research artifact, not a production OCR service.
It was trained on one collection — US federal pension files in English — and should be expected to degrade on other hands, other languages, other document genres, and other centuries. It reproduces what is written, including the spellings, abbreviations and historical language of the source; the transcription of a historical record is not an endorsement of its contents. Do not treat its output as an authoritative reading of a legal or genealogical record without human verification — at this error rate a name or a date can be wrong.