Urdu BERT (64k vocab)
A BERT-base model pretrained from scratch on Urdu text, with a custom 64,000-token WordPiece vocabulary. Trained for 20 epochs using the masked language modeling (MLM) objective.
Model Details
- Architecture: BERT-base (12 layers, 768 hidden size, 12 attention heads, 3,072 intermediate size, max position embeddings 512, type vocab size 2)
- Vocabulary size: 64,000 (custom Urdu WordPiece tokenizer, trained from scratch
with the HuggingFace
BertWordPieceTokenizer, max sequence length 512) - Pretraining objective: Masked Language Modeling (MLM) only, masking probability 0.15 — no next-sentence prediction (NSP)
- Training epochs: 20
- Language: Urdu (
ur)
Training hyperparameters: per-device batch size 24, gradient accumulation
steps 16 (effective batch size 384), learning rate 1e-4, weight decay 0.01,
warmup steps 10,000, FP16 precision, seed 42. Trained with the HuggingFace
Trainer API on a single NVIDIA H100 GPU for approximately 3–4 days.
UrduBERT was validated on two downstream tasks (Named Entity Recognition and Sentiment Analysis), outperforming multilingual BERT (mBERT) on both — detailed results will be released with the accompanying repository.
Intended Uses & Limitations
This is a base pretrained model — it has not been fine-tuned on any downstream task (e.g. classification, NER, question answering). It is intended to be used as a starting point for fine-tuning on Urdu NLP tasks, or for masked-token prediction / contextual embedding extraction as-is.
If you load this model with AutoModel (rather than AutoModelForMaskedLM), you'll
see a warning that pooler.dense.weight / pooler.dense.bias are newly initialized.
This is expected: the pooler layer was never trained (MLM pretraining does not train
it), so it holds random weights until fine-tuned on a downstream task. Do not rely on
pooler_output without fine-tuning first — use last_hidden_state instead for
embeddings.
How to Use
from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline
model_id = "farahadeeba/urdu-bert-64k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
results = fill_mask("میں [MASK] کھیل رہا ہوں")
for r in results:
print(f"{r['token_str']:15s} score={r['score']:.4f}")
Training Data
The pretraining corpus was compiled from multiple Urdu text sources:
- OSCAR 2019 — the Urdu portion of the OSCAR (Open Super-large Crawled ALMAnaCH coRpus) corpus, filtered from Common Crawl.
- GitHub repositories — publicly available Urdu text datasets.
- Kaggle datasets — publicly available Urdu text datasets.
Sentence-level deduplication was applied across the combined corpus, yielding a final corpus of approximately 5.8 GB of Urdu text. The corpus was split into 5 training files and 1 validation file of approximately 1.16 GB each (~5:1 train/validation ratio).
Citation
This model was developed as part of the following paper :
Farah Adeeba and Miriam Butt. Contextual Embedding Evidence for Main–Light Verb Distinctions in Urdu.
@unpublished{adeeba_butt_urdu_light_verbs,
title = {Contextual Embedding Evidence for Main{\textendash}Light Verb Distinctions in Urdu},
author = {Adeeba, Farah and Butt, Miriam},
year = {2026}
}
License
Apache 2.0