HmarBERT (Proof of Concept)
HmarBERT is an early proof-of-concept model for the Hmar language (hmr, ISO 639-3).
We adapted robzchhangte/MizBERT by swapping 303 cognate words in the vocabulary and fine-tuning on ~101k Hmar sentences with whole-word masking.
It worked well as a feasibility test, reaching an evaluation loss of 2.48 (perplexity 12.0). However, because it started from a Mizo base, the model still leans toward Mizo sentence patterns on less common words. For a completely clean, from-scratch model with zero Mizo priors, check out HmarBERT-mini.
Model Details
- Base model:
robzchhangte/MizBERT(BERT-base, 110M params) - Vocabulary: 30,522 tokens with 303 in-place Mizo $\rightarrow$ Hmar cognate swaps
- Training data:
hmar-heritage-org/dolma-pretrain(101,867 sentences, ~2.5M words) - Training setup: 20 epochs across 3 stages using whole-word masking
Benchmark Results
| Model / Stage | Curriculum | Eval Loss | Perplexity | Notes |
|---|---|---|---|---|
| MizBERT Base | Raw MizBERT (zero-shot) | 2.71 | 15.05 | Before adaptation |
| HmarBERT Stage 2 | 6 epochs (15% WWM, 303 swaps) | 2.55 | 12.78 | After vocab swap |
| HmarBERT Stage 3 | 8 epochs (dual cycle WWM) | 2.48 | 12.00 | Best checkpoint |
Interactive Demo
Try the model in your browser: Dolma — Powered by HmarBERT
Usage
from transformers import pipeline
fill_mask = pipeline("fill-mask", model="azinamotoe/HmarBERT")
result = fill_mask("Hrietthiem [MASK] rel el tala, ka um dan hi!")
for r in result[:3]:
print(f"{r['token_str']:<15} (score: {r['score']:.4f}) -> {r['sequence']}")