HmarBERT-mini-causal
HmarBERT-mini-causal is an autoregressively aligned causal language model derived from azinamotoe/HmarBERT-mini (16.91M parameters), trained for the Hmar language (hmr, ISO 639-3).
While HmarBERT-mini uses bidirectional attention for masked token reconstruction, HmarBERT-mini-causal enforces a causal (lower-triangular) attention mask (is_decoder=True). It is aligned specifically for ultra-fast next-word prediction, designed to power:
- Mobile Predictive Keyboards (Android / iOS keyboard engines)
- Termux / CLI Autocomplete (instant local suggestions with low memory overhead)
Model Architecture & Specs
- Base Architecture: 4-layer BERT decoder (
is_decoder=True, causal attention masking) - Parameters: 16.91M (~32.2 MB FP16 / ~67.6 MB FP32)
- Hidden Size: 384 dimensions, 6 attention heads, intermediate size 1536
- Vocabulary: 24,576 WordPiece tokens with full Hmar diacritics and morphology
- Sequence Context: Optimized for up to 64 tokens (ideal for keyboard typing windows)
- Inference Latency: <5ms per token on basic mobile / laptop CPUs
Causal Behavioral Alignment
The base model azinamotoe/HmarBERT-mini completed 48 epochs of bidirectional pretraining (24 epochs on sentences + 24 epochs on paragraphs).
In HmarBERT-mini-causal, the pre-trained weights are adapted into a causal decoder:
- Causal Masking: Lower-triangular attention ensures token $i$ only attends to tokens $\le i$.
- Training Data: Aligned on clean, segmented sentences from
hmar-heritage-org/sentences. - Behavioral Objective: Aligns attention projections to predict upcoming words strictly from left-to-right conversational history without future context.
Quickstart (Inference)
import torch
from transformers import BertTokenizerFast, BertLMHeadModel
MODEL_ID = "azinamotoe/HmarBERT-mini-causal"
tokenizer = BertTokenizerFast.from_pretrained(MODEL_ID)
model = BertLMHeadModel.from_pretrained(MODEL_ID)
model.eval()
def predict_next_words(text: str, top_k: int = 3):
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits[0, -1, :]
probs = torch.softmax(logits, dim=-1)
top_probs, top_indices = torch.topk(probs, k=top_k)
return [
(tokenizer.decode([idx.item()]).strip(), f"{prob.item() * 100:.1f}%")
for prob, idx in zip(top_probs, top_indices)
]
# Example predictions
print(predict_next_words("Ka lawm", top_k=3))
print(predict_next_words("Invaithai", top_k=3))
Citation & Organization
Maintained by the Hmar Heritage Foundation and Azinamotoe.
- Organization:
hmar-heritage-org - Base Pretrained Model:
azinamotoe/HmarBERT-mini