Donut finetuned on invoices

Based on Donut base model (introduced in the paper OCR-free Document Understanding Transformer by Geewok et al. and first released in this repository.

The model was trained with a few thousand of annotated invoices and non-invoices (for those the doctype will be 'Other'). They span across different countries and languages. They are always one page only. The dataset is proprietary unfortunately. Model is set to input resolution of 1280x1920 pixels. So any sample you want to try with higher dpi than 150 has no added value. It was trained for about 4 hours on a NVIDIA RTX A4000 for 20k steps with a val_metric of 0.03413819904382196 at the end. The following indexes were included in the train set:

DocType Currency DocumentDate GrossAmount InvoiceNumber NetAmount TaxAmount OrderNumber CreditorCountry

Demo space can be found here

Model description

Donut consists of a vision encoder (Swin Transformer) and a text decoder (BART). Given an image, the encoder first encodes the image into a tensor of embeddings (of shape batch_size, seq_len, hidden_size), after which the decoder autoregressively generates text, conditioned on the encoding of the encoder.

Intended uses & limitations

This model is meant as a research in how well it fares with multilanguage invoices. See my observations in the demo space.

How to use

Look at the documentation which includes code examples.