HmarBERT-mini
HmarBERT-mini is a small, 4-layer BERT model trained from scratch for the Hmar language (hmr, ISO 639-3), part of the Zo languages family.
At roughly 16.9M parameters (~32 MB in FP16), it runs quickly on laptops, phones, or basic CPUs.
Model Details
- Architecture: 4-layer BERT encoder (384 hidden size, 6 attention heads)
- Parameters: 16.91M (~32.2 MB FP16 / ~67.6 MB FP32)
- Vocabulary: 24,576 WordPiece tokens (uncased)
- Max Sequence Length: 512 tokens
- Masking: Whole-word masking (6-epoch cycles, 100% token coverage)
- Training length: 48 epochs (8 cycles)
Training
Training ran in two phases:
- Epochs 1–24: ~265k short sentences (
hmar-heritage-org/sentences) to learn basic grammar, affixes, and sentence structure. - Epochs 25–48: ~33k paragraphs (
hmar-heritage-org/paragraphs, ~3.86M words) from scripture, prose, essays, and online comments to handle full context up to 512 tokens.
Tokenizer
- Accents: Typing without accents (
tirin,van,Pathien) maps to the same tokens as formal text with accents (tîrin,vân,Pathien). - Emojis and symbols: Cleaned during preprocessing so they don't produce accidental
[UNK]tokens. - Hmar characters: Preserves all standard letters and digraphs (
ṭ,kh,th,ng,ch).
Quickstart
from transformers import pipeline
unmasker = pipeline("fill-mask", model="azinamotoe/HmarBERT-mini")
# Example 1: Prose
text1 = "Chu zan chun lung an lêng deu rieu leiin a ruolnu chu hla rikawt insaktir dingin kan [MASK] a."
print(unmasker(text1)[0]['sequence'])
# -> "...hla rikawt insaktir dingin kan ngên a."
# Example 2: Scripture
text2 = "A tîrin Pathienin hnuoi le [MASK] a siem a."
print(unmasker(text2)[0]['sequence'])
# -> "A tîrin Pathienin hnuoi le vân a siem a."
Evaluation
For test results, accuracy curves, and qualitative examples across different registers, see:
BENCHMARK.md— Evaluation report and case studiesbenchmark_metrics.json— Training loss and learning rate historybenchmark_predictions.jsonl— Model predictions on the test set
Acknowledgements
Trained on open datasets curated by the Hmar Heritage Foundation (hmar-heritage-org).