mmbert-small-vi-exam-seq-labeling

Production-grade token classification and sequence labelling model fine-tuned from jhu-clsp/mmBERT-base on the comprehensive Vietnamese Exam Sequence Labelling Dataset v2.

Designed for granular structural extraction from Vietnamese educational examinations (grades 8–12 across Math, Physics, Chemistry, Biology, History, Geography, Literature, and English), supporting both scanned and digital documents with Zero Character Mutation guarantee.

Model Architecture

• Base Encoder: jhu-clsp/mmBERT-base (Multilingual ModernBERT)
• Classification Head: EnhancedBertForTokenClassification featuring Multi-Layer Hidden State Fusion (fusing the last 4 hidden transformer layers via learned scalar weights) + Dropout + Linear projection.
• Context Length: Trained with multi-scale sliding window configurations ([512, 128], [768, 192], [1024, 256], [2048, 512]) to preserve both short question boundaries and extended reading comprehension passages.

Entity Schema (7 Core Examination Entities)

The model performs token-level sequence labelling using standard BIO tagging:

Architectural Invariants

1. Stemless Cloze Questions: Full support for cloze / fill-in questions where <question_label> directly precedes <option_label> without an explicit <stem>.
2. Zero Character Mutation Guarantee: Sequence labeling and XML reconstruction wrap tags directly around input substrings with 0% character deletion, alteration, or hallucination.

Evaluation Results (Gold Benchmark Test Set)

Evaluated on the strictly isolated 4-tier Gold Benchmark (63 pristine real exam documents, ~458k tokens):

Checkpoint synced at Epoch 2/2.

Quickstart & Usage

1. Installation

pip install torch transformers huggingface_hub

2. Python Inference

import torch
from transformers import AutoTokenizer
from model.module.head import EnhancedBertForTokenClassification
from model.inference.predict import predict_text, load_label_mapping

repo_id = "daominhwysi/mmbert-small-vi-exam-seq-labeling"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
tag_to_id, id_to_tag = load_label_mapping(repo_id)

model = EnhancedBertForTokenClassification.from_pretrained(
    repo_id,
    num_labels=len(tag_to_id),
    id2label=id_to_tag,
    label2id=tag_to_id
).to(device)

text = """
Read the following passage and mark the letter A, B, C or D on your answer sheet.
Digital tools help modern students collaborate more effectively across borders.
Question 1: What do digital tools help students do?
A. Play games
B. Collaborate effectively
C. Avoid homework
D. Work alone
"""

res = predict_text(text, model=model, tokenizer=tokenizer, id_to_tag=id_to_tag, device=device)
print(res["xml_text"])

Output:

<section>Read the following passage and mark the letter A, B, C or D on your answer sheet.</section>
<stimulus>Digital tools help modern students collaborate more effectively across borders.</stimulus>
<question_label>Question 1:</question_label> <stem>What do digital tools help students do?</stem>
<option_label>A.</option_label> <option_text>Play games</option_text>
<option_label>B.</option_label> <option_text>Collaborate effectively</option_text>
<option_label>C.</option_label> <option_text>Avoid homework</option_text>
<option_label>D.</option_label> <option_text>Work alone</option_text>

Training Details

• Dataset: daominhwysi/synthetic-seq-labelling-vi-exam-v2 (776 documents, 5.2M tokens, 90k multi-scale sliding chunks).
• Optimizer: AdamW (Learning Rate: 3e-05) with Linear Warmup and Cosine Decay.
• Batch Size: 4
• Sequence Length: None
• License: Apache 2.0