ModernLayoutLM

ModernLayoutLM is an attempt to produce a long-context (like ModernBERT) bidirectional encoder that incorporates layout information into its training (like LayoutLM). ModernLayoutLM uses the same recipe as Xu et al. (2020) in the creation of LayoutLMv1, where the major difference is swapping BERT with ModernBERT (Warner et al., 2024).

Intended use

The model is a pre-trained checkpoint that can be fine-tuned to layout understanding tasks, like named entity recognition or document classification.

from transformers import AutoModelForMaskedLM, AutoProcessor

path = "kkuroma/ModernLayoutLM"
model = AutoModelForMaskedLM.from_pretrained(path, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(path, trust_remote_code=True)

batch = processor(words=["Total", "42.00"], boxes=[[10, 10, 60, 20], [70, 10, 140, 20]], max_length=512, return_tensors="pt")
model(**batch)

• bbox holds one (x0, y0, x1, y1) per word, with the origin at the top left of the page.
• Coordinates are normalized to [0, 1000] by the page width and height, not given in pixels.
• Every subword of a word takes the word's box, which the bundled processor handles.
• padding defaults to max_length, which without an explicit length is the model's own 8192.
• bbox=None skips the layout sum and the model runs as plain ModernBERT, while all-zero boxes put every word at the origin.
• Attention defaults to SDPA; attn_implementation="flash_attention_2" with flash-attn installed is faster and lighter on long documents, see Inference cost.

Training data

ModernLayoutLM is pretrained on about 11M pages of English business and scientific documents, each read once, in two stages.

Pages carrying a Bates number from the FUNSD or RVL-CDIP evaluation splits are dropped from both stages, since IDL and those benchmarks share IIT-CDIP.

Training procedure

ModernLayoutLM is pretrained with the masked visual-language model of LayoutLMv1 (Xu et al., 2020). Four 2D position embedding tables, where x0 and x1 share table X and y0 and y1 share table Y, are summed into the token embeddings before the embedding LayerNorm. Text is masked and boxes are left intact, so recovering a masked token rewards reading its position on the page. The four tables start at zero, so the model starts numerically identical to ModernBERT. The number of iterations and the duration are set by hardware limitations, as every step ran on a single RTX 3090.

Evaluation

FUNSD

• Source: Jaume et al. (2019), read from nielsr/funsd
• Released: 2019
• Size: 199 scanned forms, 149 train and 50 test
• Task: entity labeling over header, question and answer, as 7 BIO labels

CORD

• Source: Park et al. (2019), read from naver-clova-ix/cord-v2
• Released: 2019
• Size: 1,000 Indonesian receipts, 800 train, 100 validation and 100 test
• Task: receipt field labeling over 29 categories such as menu name, price and total, as 59 BIO labels

SROIE

• Source: the ICDAR 2019 Competition on Scanned Receipt OCR and Information Extraction (Huang et al., 2019), read from Ssunbell/SROIE_layoutlmv2_sequence
• Released: 2019
• Size: 973 receipts, 594 train, 32 validation and 347 test
• Task: key information extraction over company, date, address and total, as 9 BIO labels

RealKIE

• Source: Townsend et al. (2024), the FCC Invoices, NDA and Charities sets, read from the Zenodo release with its own OCR
• Released: 2024
• Size: long multi-page documents, 38% to 55% of them past 4096 tokens
• Task: key information extraction labeled as character spans over the OCR text

Results

Each column is one model fine-tuned on identical data.

• ModernLayoutLM: this model, fine-tuned with boxes.
• LayoutLMv1: microsoft/layoutlm-base-uncased, fine-tuned with boxes.
• ModernLayoutLM (no-pretrain): the same architecture fine-tuned with boxes straight from ModernBERT, skipping the layout pretraining.
• ModernLayoutLM (no-bbox): this model fine-tuned with every box at the origin.
• ModernBERT: answerdotai/ModernBERT-base fine-tuned on text alone.

ModernLayoutLM performs comparably to LayoutLMv1, but natively supports long context.

Inference cost

Citation

Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., & Zhou, M. (2020). LayoutLM: Pre-training of Text and Layout for Document Image Understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 1192-1200). https://doi.org/10.1145/3394486.3403172

Warner, B., Chaffin, A., Clavié, B., Weller, O., Hallström, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., Cooper, N., Adams, G., Howard, J., & Poli, I. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. Preprint.

@inproceedings{xu2020layoutlm,
  title={LayoutLM: Pre-training of Text and Layout for Document Image Understanding},
  author={Xu, Yiheng and Li, Minghao and Cui, Lei and Huang, Shaohan and Wei, Furu and Zhou, Ming},
  booktitle={Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining},
  pages={1192--1200},
  year={2020},
  publisher={ACM},
  doi={10.1145/3394486.3403172}
}

@misc{warner2024smarterbetterfasterlonger,
  title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author={Benjamin Warner and Antoine Chaffin and Benjamin Clavi{\'e} and Orion Weller and Oskar Hallstr{\"o}m and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
  year={2024}
}

License

Apache 2.0, the license ModernBERT is released under, see LICENSE.