Arch-L3869-PageClassification

Model Details

Model Description

This is a Greek text classification model for categorizing document pages into 18 different classes. The model was trained using a two-phase approach:

1. Phase 1 (Contrastive Learning): Further pre-training of the base BERT model using Supervised Contrastive Learning (SCL) to create better document embeddings.
2. Phase 2 (Classification): Fine-tuning with Asymmetric Loss for handling class imbalance.

• Developed by: Archeiothiki S.A. - AI Services Team
• Model type: BertForSequenceClassification
• Language(s): Greek (el)
• Finetuned from model: nlpaueb/bert-base-greek-uncased-v1

Model Architecture

• Base Model: nlpaueb/bert-base-greek-uncased-v1
• Pruned Layers: [0, 2, 4, 6, 8, 11] (6 layers kept for efficiency)
• Hidden Size: 768
• Attention Heads: 12
• Max Position Embeddings: 512
• Vocab Size: 35,000

Uses

Direct Use

This model classifies document pages (text extracted via OCR) into one of 18 categories:

How to Get Started with the Model

Prerequisites

pip install transformers torch

Preprocessing Function (Required!)

⚠️ IMPORTANT: This preprocessing MUST be applied to all texts before inference. The model was trained with this preprocessing.

import re
import unicodedata

# Same symbols removed during training
SYMBOLS_TO_REMOVE = r"[`~!@#$%^&*()\-+=\[\]{\}/?><,\'\":;|»«§°·¦ʼ¬£€©΄´\\…\n]"

def strip_accents_and_lowercase(text: str) -> str:
    """Remove accents and convert to lowercase."""
    return "".join(
        c for c in unicodedata.normalize("NFD", text)
        if unicodedata.category(c) != "Mn"
    ).lower()

def clean_text(text: str, symbols_to_remove: str | None = None) -> str:
    """
    Main preprocessing function.

    Steps:
        1. Remove special symbols
        2. Collapse multiple dots into single dot
        3. Remove accents + lowercase
        4. Normalize whitespace
    """
    if symbols_to_remove:
        text = re.sub(symbols_to_remove, " ", text)

    text = re.sub(r"\.{2,}", ". ", text)
    text = strip_accents_and_lowercase(text)
    text = re.sub(r"\s+", " ", text).strip()
    return text

def preprocess_text(text: str) -> str:
    return clean_text(text, symbols_to_remove=SYMBOLS_TO_REMOVE)

Inference Code Snippet (includes preprocessing + dummy strings)

import json
import re
import unicodedata
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Preprocessing (REQUIRED!)
SYMBOLS_TO_REMOVE = r"[`~!@#$%^&*()\-+=\[\]{\}/?><,\'\":;|»«§°·¦ʼ¬£€©΄´\\…\n]"

def strip_accents_and_lowercase(text: str) -> str:
    return "".join(
        c for c in unicodedata.normalize("NFD", text)
        if unicodedata.category(c) != "Mn"
    ).lower()

def clean_text(text: str, symbols_to_remove: str | None = None) -> str:
    if symbols_to_remove:
        text = re.sub(symbols_to_remove, " ", text)
    text = re.sub(r"\.{2,}", ". ", text)
    text = strip_accents_and_lowercase(text)
    text = re.sub(r"\s+", " ", text).strip()
    return text

def preprocess_text(text: str) -> str:
    return clean_text(text, symbols_to_remove=SYMBOLS_TO_REMOVE)

# Load model and tokenizer
MODEL_PATH = "path/to/model"
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_PATH)
model.eval()

# Load label mapping
with open(f"{MODEL_PATH}/id2label.json", "r", encoding="utf-8") as f:
    id2label = json.load(f)

# Dummy texts (examples)
texts = [
    "ΔΕΛΤΙΟ ΑΣΤΥΝΟΜΙΚΗΣ ΤΑΥΤΟΤΗΤΑΣ ΠΑΠΑΔΟΠΟΥΛΟΣ ΙΩΑΝΝΗΣ",
    "ΕΝΤΥΠΟ Ε1 ΔΗΛΩΣΗ ΦΟΡΟΛΟΓΙΑΣ ΕΙΣΟΔΗΜΑΤΟΣ 2024",
]

# Preprocess texts
preprocessed_texts = [preprocess_text(t) for t in texts]

# Tokenize
inputs = tokenizer(
    preprocessed_texts,
    truncation=True,
    padding="max_length",
    max_length=512,
    return_tensors="pt"
)

# Inference
with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits
    probabilities = torch.sigmoid(logits)  # Multi-label sigmoid
    predictions = probabilities.argmax(dim=1)

# Get labels
for i, pred in enumerate(predictions):
    label = id2label[str(pred.item())]
    confidence = probabilities[i][pred].item()
    print(f"Text: {texts[i][:50]}...")
    print(f"Prediction: {label} (confidence: {confidence:.4f})")
    print()

Expected Output

Text: ΔΕΛΤΙΟ ΑΣΤΥΝΟΜΙΚΗΣ ΤΑΥΤΟΤΗΤΑΣ ΠΑΠΑΔΟΠΟΥΛΟΣ ΙΩΑΝΝΗΣ...
Prediction: AA_ID_Card (confidence: 0.9842)

Text: ΕΝΤΥΠΟ Ε1 ΔΗΛΩΣΗ ΦΟΡΟΛΟΓΙΑΣ ΕΙΣΟΔΗΜΑΤΟΣ 2024...
Prediction: AA_INCOME_TAX_RETURN_-_E1 (confidence: 0.9567)

Training Details

Training Data

• Dataset: Internal annotated document dataset
• Total Samples: ~6,600 (train + validation)
• Test Samples: 1,336
• Classes: 18 (imbalanced distribution)
• Largest Class: Other (571 test samples, ~43%)
• Smallest Class: AA_LEGAL_ENTITY_MINUTES (7 test samples, ~0.5%)

Training Procedure

Phase 1: Contrastive Learning

• Base Model: nlpaueb/bert-base-greek-uncased-v1
• Loss Function: Supervised Contrastive Loss (SCL)
• Epochs: 200
• Learning Rate: 2e-5
• Batch Size: 32
• Layer Pruning: Kept layers [0, 2, 4, 6, 8, 11]

Phase 2: Classification

• Base Model: Output of Phase 1 (26_01_2026_15_00_12)
• Loss Function: Asymmetric Loss (gamma=4)
• Epochs: 50
• Learning Rate: 1e-4
• Batch Size: 32
• Gradient Accumulation: 2
• Warmup Ratio: 0.1
• LR Scheduler: Cosine
• Oversampling: BB_Other_Documents (x2)

Framework Versions

• Python: 3.9.0
• PyTorch: 2.x
• Transformers: 4.38.2
• Datasets: 2.x

Evaluation Results

Overall Metrics (Test Set: 1,336 samples)

Per-Class Performance

Key Performance Highlights

• ✅ Other class: F1=0.95 (excellent handling of the majority class)
• ✅ BB_Other_Documents: F1=0.72 (best among all trained models for this rare class)
• ✅ High-confidence classes: AA_ID_Card, AA_Certificate, AA_Legal_Entity_Minutes all achieve 1.00 F1
• ⚠️ Lower performance: AA_LEGAL_ENT_CERTIFICATE (F1=0.79) - needs more training data

Model Files

Model Card Authors

AI Services Team - Archeiothiki S.A.

Model Card Contact

Internal use only.