Image classification using fine-tuned ViT - for historical :bowtie: documents sorting

Goal: solve a task of archive page images sorting (for their further content-based processing)

Scope: Processing of images, training and evaluation of ViT model, input file/directory processing, class 🏷️ (category) results of top N predictions output, predictions summarizing into a tabular format, HF 😊 hub support for the model

Versions 🏁

There are currently 2 version of the model available for download, both of them have the same set of categories, but different data annotations. The latest v4.3 is considered to be default and can be found in the main branch of HF 😊 hub ¹ 🔗

Model description 📇

🔲 Fine-tuned model repository: vit-historical-page ¹ 🔗

🔳 Base model repository:

• Google's vit-base-patch16-224, vit-base-patch16-384, and vit-large-patch16-284 ^{2 3 4} 🔗
• timm's regnety_160.swag_ft_in1k, efficientnetv2_s.in21k, efficientnetv2_m.in21k_ft_in1k, and efficientnetv2_l.in21k_ft_in1k ^{5 6 7 8} 🔗

Data 📜

The dataset is provided under Public Domain license, and consists of 48,499 PNG images of pages from 37,328 archival documents. The source image files and their annotation can be found in the LINDAT repository ⁹ 🔗.

Manual ✍️ annotation was performed beforehand and took some time ⌛, the categories 🪧 tabulated below were formed from different sources of the archival documents originated in the 1920-2020 years span.

The table above shows category distribution for different model versions, where the last column (Dataset 3) corresponds to the latest vX.3 models data, which actually used 14,000 pages of TEXT category, while other columns cover all the used samples - specifically 80% as training 💪, and 10% each as development and test 🏆 sets. The early model version used 90% of the data as training 💪 and the remaining 10% as both development and test 🏆 set due to the lack of annotated (manually classified) pages.

[!NOTE] Disproportion of the categories 🪧 in both training and evaluation data is NOT intentional, but rather a result of the source data nature.

Training set of the model: 8950 images for v2.0

Training set of the model: 10745 images for v2.1

Training set of the model: 14565 images for v2.X

Training set of the model: 38625 images for vX.3

Plus, the test sets:

Evaluation set: 1586 images (taken from v2.2 annotations)

Evaluation set: 4823 images (for vX.3 models)

Categories 🏷️

Data preprocessing

During training the following transforms were applied randomly with a 50% chance:

• transforms.ColorJitter(brightness 0.5)
• transforms.ColorJitter(contrast 0.5)
• transforms.ColorJitter(saturation 0.5)
• transforms.ColorJitter(hue 0.5)
• transforms.Lambda(lambda img: ImageEnhance.Sharpness(img).enhance(random.uniform(0.5, 1.5)))
• transforms.Lambda(lambda img: img.filter(ImageFilter.GaussianBlur(radius=random.uniform(0, 2))))

Training Hyperparameters

• eval_strategy "epoch"
• save_strategy "epoch"
• learning_rate 5e-5
• per_device_train_batch_size 8
• per_device_eval_batch_size 8
• num_train_epochs 3
• warmup_ratio 0.1
• logging_steps 10
• load_best_model_at_end True
• metric_for_best_model "accuracy"

Results 📊

v2.2 Evaluation set's accuracy (Top-1): 97.54%

v3.2 Evaluation set's accuracy (Top-1): 96.49%

v5.2 Evaluation set's accuracy (Top-1): 97.73%

v1.2 Evaluation set's accuracy (Top-1): 97.73%

v4.2 Evaluation set's accuracy (Top-1): 97.86%

v1.3 Evaluation set's accuracy (Top-1): 98.83%

v2.3 Evaluation set's accuracy (Top-1): 98.79%

v3.3 Evaluation set's accuracy (Top-1): 98.92%

v4.3 Evaluation set's accuracy (Top-1): 98.16%

v5.3 Evaluation set's accuracy (Top-1): 99.12%

v6.3 Evaluation set's accuracy (Top-1): 98.79%

Result tables

• v2.2 Manually ✍ checked evaluation dataset results (TOP-1): model_TOP-3_EVAL.csv 🔗

• v2.2 Manually ✍ checked evaluation dataset results (TOP-3): model_TOP-3_EVAL.csv 🔗

• v3.2 Manually ✍ checked evaluation dataset results (TOP-1): model_TOP-3_EVAL.csv 🔗

• v3.2 Manually ✍ checked evaluation dataset results (TOP-3): model_TOP-3_EVAL.csv 🔗

• v5.2 Manually ✍ checked evaluation dataset results (TOP-1): model_TOP-3_EVAL.csv 🔗

• v5.2 Manually ✍ checked evaluation dataset results (TOP-3): model_TOP-3_EVAL.csv 🔗

• v1.2 Manually ✍ checked evaluation dataset results (TOP-1): model_TOP-3_EVAL.csv 🔗

• v1.2 Manually ✍ checked evaluation dataset results (TOP-3): model_TOP-3_EVAL.csv 🔗

• v4.2 Manually ✍ checked evaluation dataset results (TOP-1): model_TOP-3_EVAL.csv 🔗

• v4.2 Manually ✍ checked evaluation dataset results (TOP-3): model_TOP-3_EVAL.csv 🔗

• v1.3 Manually ✍ checked evaluation dataset (TOP-1): model_TOP-1_EVAL.csv 📎

• v2.3 Manually ✍ checked evaluation dataset (TOP-1): model_TOP-1_EVAL.csv 📎

• v3.3 Manually ✍ checked evaluation dataset (TOP-1): model_TOP-1_EVAL.csv 📎

• v4.3 Manually ✍ checked evaluation dataset (TOP-1): model_TOP-1_EVAL.csv 📎

• v5.3 Manually ✍ checked evaluation dataset (TOP-1): model_TOP-1_EVAL.csv 📎

• v6.3 Manually ✍ checked evaluation dataset (TOP-1): model_TOP-1_EVAL.csv 📎

• Ensemble (5-model) evaluation dataset predictions (TOP-1): 20260530-1234_BEST_5_models_TOP-1.csv 📎

• Ensemble (5-model) averaged evaluation dataset (TOP-3): averaged_results_TOP-3_sorted.csv 📎

Contacts 📧

For support write to 📧 lutsai.k@gmail.com 📧

Official repository: UFAL ¹⁰

Preprint 📖

For the full research background, check out our paper on arXiv: Page image classification for content-specific data processing

It covers everything from raw data exploration and dataset construction 🗂️, through benchmarking of multiple image classification approaches (Random Forest, EfficientNetV2, RegNetY, DiT, ViT, and CLIP), to system architecture and real-world results on historical collections from Prague ⛪ and Brno 🏛️.

Acknowledgements 🙏

• Developed by UFAL ¹¹ 👥
• Funded by ATRIUM ¹² 💰
• Shared by ATRIUM ¹² & UFAL ¹¹
• Model type:
  • fine-tuned ViT with a 224x224 ² 🔗 or 384x384 ^{3 4} 🔗 resolution size
  • fine-tuned EffNetV2 with a 300x300 ⁶ 🔗 or 384x384 ⁸ 🔗 resolution size

©️ 2022 UFAL & ATRIUM