MaithiliBERT: First RoBERTa Model for the Maithili Language
MaithiliBERT is a RoBERTa-base language model (124M parameters) pretrained from scratch on MaithiliCorpus — the largest open Maithili text corpus comprising 39.8M clean, LID-filtered, and deduplicated words. This is the first RoBERTa ever trained specifically for Maithili, a low-resource Indo-Aryan language spoken by approximately 50 million people primarily in Bihar, India and Nepal.
Model Architecture
| Property | Value |
|---|---|
| Architecture | RoBERTa-base |
| Parameters | 124,493,648 |
| Vocabulary | Byte-level BPE, 50,000 tokens |
| Context Window | 512 tokens |
| Hidden Size | 768 |
| Intermediate Size | 3072 |
| Attention Heads | 12 |
| Hidden Layers | 12 |
| Training Data | MaithiliCorpus gold_deduped (47,882 documents, 39.8M words) |
| Training Steps | 14,210 (10 epochs) |
| Effective Batch Size | 32 |
| Optimizer | AdamW (lr=5e-4) |
| Precision | FP16 |
| License | CC-BY-4.0 |
Tokenizer
The tokenizer is a byte-level BPE trained from scratch on MaithiliCorpus with a vocabulary of 50,000 tokens — 60% smaller than mBERT (119K) and 80% smaller than XLM-R/IndicBERTv2 (250K). This reduces embedding parameters from 192M to 38.4M, shrinking the embedding layer from 61% to 31% of total model parameters.
| Property | Value |
|---|---|
| Algorithm | Byte-level BPE |
| Vocabulary Size | 50,000 |
| Training Data | MaithiliCorpus (39.8M words) |
| Training | SentencePiece, --model_type=bpe |
| Special Tokens | <s>, </s>, <unk>, <pad>, <mask> |
| UNK Rate (on Maithili) | 0.00% |
Acknowledgments
The authors gratefully acknowledge AnK Komputing for providing the computational resources and infrastructure used to conduct the simulations/analyses presented in this work. We thank the team at AnK Komputing for their technical support and access to their high-performance computing services.
Intended Use
- Masked language modeling (fill-mask)
- Fine-tuning for text classification, NER, and other NLU tasks
- Feature extraction for downstream Maithili NLP
Usage
from transformers import pipeline
fill = pipeline("fill-mask", model="AiventraLab/maithili-roberta")
results = fill("भगवान् <mask>क ध्यान")