KhaBERT (Pretrained - 100 Epochs)
KhaBERT is a BERT-based language model pretrained from scratch on monolingual Khasi text using the Masked Language Modeling (MLM) objective. This checkpoint represents the model after completing 300,000 training steps (100 epochs).
Model Details
- Model Architecture: BERT (
BertForMaskedLM) - Language: Khasi (
kha) - Tokenizer: Custom WordPiece Tokenizer (
vocab_size: 30,000) - Hidden Size: 768
- Intermediate Size: 3072
- Number of Attention Heads: 12
- Number of Hidden Layers: 12
- Max Position Embeddings: 512 (Tokenized at 128 during pretraining)
Training Data
The model was pretrained on a custom Khasi monolingual corpus compiled from news and local text sources:
- Total Training Samples: 191,937 sentences
- Evaluation Samples: 10,102 sentences
- Split Ratio: 95% Train / 5% Validation
Training Procedure
- Objective: Masked Language Modeling (MLM) with 15% masking probability
- Epochs: 100
- Batch Size: 64 per device
- Precision: Mixed Precision (
fp16 = True) - Total Steps: 300,000
Evaluation Results
Performance evaluated on the held-out validation set (10,102 samples) at the final 100-epoch checkpoint:
| Metric | Score |
|---|---|
| Evaluation Loss | 1.1922 |
| MLM Accuracy | 74.47% |
| Perplexity | 3.2943 |
How to Use
You can use this model directly with a pipeline for masked language modeling:
from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline
model_name = "YOUR_USERNAME/khabert-pretrained"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
results = fill_mask("Leit hiar thma jur biang ka [MASK] sha ka nongbah.")
for result in results:
print(f"Token: {result['token_str']} | Score: {result['score']:.4f}")