LayoutLMv3 fine-tuned on CORD (Receipt Understanding)

This model is a fine-tuned version of microsoft/layoutlmv3-base on the CORD (Consolidated Receipt Dataset) for document field extraction (token classification).

Instead of just extracting raw text like traditional OCR, this model understands the structure of a receipt/invoice and classifies each word into a specific field (item name, price, quantity, tax, total, etc.), combining text, layout (bounding boxes), and the document image itself.

Model Details

• Base model: microsoft/layoutlmv3-base
• Task: Token Classification (NER-style, BIO tagging)
• Training data: CORD — 800 train / 100 dev / 100 test receipt images
• Number of labels: 54 (see full label list below)
• Framework: 🤗 Transformers (LayoutLMv3ForTokenClassification)

Intended Uses & Limitations

Intended use

Extracting structured fields from receipt-style documents (item names, prices, quantities, subtotal, tax, total, payment method) as part of a document understanding / data extraction pipeline feeding a database or RAG system.

Limitations

• Trained on only 800 Indonesian retail receipts — may not generalize well to invoices, contracts, forms, or documents in other languages/layouts without additional fine-tuning.
• Requires word-level bounding boxes as input. If your OCR engine returns line-level boxes (e.g. some PaddleOCR versions), you'll need to split lines into individual words with estimated per-word boxes before inference — otherwise BIO tag predictions may be inconsistent.
• OCR quality (blurry scans, unusual fonts) directly impacts extraction accuracy, since this model relies on OCR output as input (it does not read the image text itself).

How to Use

from transformers import LayoutLMv3ForTokenClassification, LayoutLMv3Processor
from PIL import Image
import torch

model = LayoutLMv3ForTokenClassification.from_pretrained("your-username/layoutlmv3-cord-invoice")
processor = LayoutLMv3Processor.from_pretrained("your-username/layoutlmv3-cord-invoice", apply_ocr=False)

image = Image.open("receipt.jpg").convert("RGB")

# words and boxes must come from your own OCR step (e.g. PaddleOCR),
# with boxes normalized to a 0-1000 scale
words = [...]          # list[str]
boxes = [...]           # list[[x0, y0, x1, y1]], normalized 0-1000

encoding = processor(
    image, words, boxes=boxes,
    truncation=True, padding="max_length", max_length=512, return_tensors="pt"
)

with torch.no_grad():
    outputs = model(**encoding)

predictions = outputs.logits.argmax(-1).squeeze().tolist()
labels = [model.config.id2label[p] for p in predictions]

⚠️ This model does not include an OCR engine. You must run OCR (e.g. PaddleOCR, Tesseract) first to obtain words and boxes, since apply_ocr=False.

Label List

The model uses BIO tagging over the following CORD field categories:

MENU.NM, MENU.CNT, MENU.PRICE, MENU.UNITPRICE, MENU.DISCOUNTPRICE,
MENU.SUB_NM, MENU.SUB_CNT, MENU.SUB_PRICE, MENU.SUB_UNITPRICE, MENU.ETC,
MENU.SUB_ETC, MENU.NUM, MENU.VATYN,
SUB_TOTAL.SUBTOTAL_PRICE, SUB_TOTAL.TAX_PRICE, SUB_TOTAL.SERVICE_PRICE,
SUB_TOTAL.DISCOUNT_PRICE, SUB_TOTAL.ETC, SUB_TOTAL.OTHERSVC_PRICE,
TOTAL.TOTAL_PRICE, TOTAL.CASHPRICE, TOTAL.CHANGEPRICE, TOTAL.CREDITCARDPRICE,
TOTAL.EMONEYPRICE, TOTAL.MENUQTY_CNT, TOTAL.MENUTYPE_CNT, TOTAL.TOTAL_ETC,
VOID_MENU.NM, O

(each prefixed with B- / I- per standard BIO tagging — 54 labels total)

Training Procedure

1. Parsed raw CORD JSON annotations (valid_line, dontcare) into word-level (text, bbox, label) triples, preserving original word order within each entity for correct BIO sequencing.
2. Normalized bounding boxes to a 0–1000 scale (clamped to avoid out-of-range values).
3. Tokenized with LayoutLMv3Processor, aligning sub-word tokens to their parent word's label (-100 for non-first sub-tokens, per standard token classification practice).
4. Fine-tuned LayoutLMv3ForTokenClassification using the 🤗 Trainer API.

Training hyperparameters

• Learning rate: 2e-5
• Batch size: 4 (train and eval)
• Epochs: 15
• Evaluation strategy: per epoch

Evaluation

Evaluated using seqeval (entity-level Precision, Recall, F1). See training logs for per-label breakdown.

License

The CORD dataset used for fine-tuning is licensed under CC BY 4.0.