BnLM-F: Bangla Language Model (135M)

BnLM-F is a 135M-parameter Bangla-specific pretrained language model, one of three models in the BnLM suite (Bengali Language Model) introduced alongside the BLUGE benchmark and B-CORE pretraining corpus. BnLM-F is among the smallest Bangla language models available, while ranking among the most accurate and computationally efficient — delivering state-of-the-art results at a fraction of the size and compute of multilingual alternatives. BnLM models are pretrained from scratch on Bangla text using the Masked Language Modeling (MLM) objective, and are designed for efficient, low-resource NLP without relying on large multilingual models.

This is a base (pretrained) model. It has been trained only with the MLM objective and has not been fine-tuned for any downstream task. It works out of the box for masked-language-modeling and feature extraction. For classification, NER, and similar tasks, you must fine-tune it first — see Fine-tuning below.

See the BLUGE collection for the full release — evaluation tasks, pretraining corpus, tokenizers, and all three BnLM variants (BnLM-F, BnLM-M, BnLM-C).

Model Details

Despite its compact size, BnLM-F achieves state-of-the-art results across BLUGE and four cross-lingual/Indic benchmarks, while requiring 44–91% fewer FLOPs and ~2x faster inference than competitive multilingual baselines. Full training configuration and hyperparameters are available in the paper.

Usage

Masked language modeling (works out of the box)

This is what the pretrained model does directly, with no additional training:

from transformers import pipeline

fill = pipeline("fill-mask", model="nahid-hub/BnLM-F-135m")

# The mask token is read from the tokenizer, so this works regardless of
# how the special token is spelled in this model's custom vocabulary.
mask = fill.tokenizer.mask_token
print(fill(f"আমি বাংলায় গান {mask}।"))

Loading the encoder (feature extraction or custom heads)

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("nahid-hub/BnLM-F-135m")
model = AutoModel.from_pretrained("nahid-hub/BnLM-F-135m")

inputs = tokenizer("খাবারটা অসাধারণ ছিল।", return_tensors="pt")
outputs = model(**inputs)          # last_hidden_state, etc.

To access the MLM head instead, use AutoModelForMaskedLM.from_pretrained("nahid-hub/BnLM-F-135m").

Fine-tuning for downstream tasks

To use BnLM-F for classification (e.g. sentiment on BLUGE-TSC), load it with a task head and train it:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("nahid-hub/BnLM-F-135m")
model = AutoModelForSequenceClassification.from_pretrained(
    "nahid-hub/BnLM-F-135m",
    num_labels=3,   # e.g. 0: neutral, 1: positive, 2: negative
)

# NOTE: The classification head above is RANDOMLY INITIALIZED. Its predictions
# are meaningless until you fine-tune the model on labeled data. Loading this
# checkpoint will report `classifier`/`pre_classifier` weights as "missing"
# (newly initialized) and the MLM-head weights as "unexpected" (discarded) —
# both are expected for a base model and are not errors.
#
# Fine-tune with the Trainer API or your own loop, then save/push the result:
#     model.save_pretrained("BnLM-F-135m-tsc")
#     model.push_to_hub("nahid-hub/BnLM-F-135m-tsc")   # requires a write token
#
# Only after fine-tuning should you run inference (logits -> argmax -> label).

See the Hugging Face text-classification guide for a complete training example.

Training Data

Pretrained exclusively on B-CORE, a 16.5M-document, 4.32-billion-token Bangla corpus built via a reproducible multi-stage pipeline achieving a 22.4% reduction in corpus volume through quality filtering and cross-corpus deduplication.

Related Models

• BnLM-C — 66M parameters, compact variant with 30.5K-vocabulary tokenizer

License

Released under CC BY 4.0.

Citation

If you use this model, please cite:

@ARTICLE{BnLM-BLUGE-B-CORE,
  author={Hossain, Nahid and Faisal Kabir, Md.},
  journal={IEEE Access}, 
  title={Efficient Monolingual Pretraining in Low-Resource Settings Through Morphology-Aware Tokenization, Principled Corpus Denoising, and Benchmark-Driven Evaluation}, 
  year={2026},
  volume={14},
  number={},
  pages={91979-92003},
  keywords={Modeling;Multilingual;Training;Cleaning;Vocabulary;Labeling;Tokenization;Computational linguistics;Pipelines;Conferences;B-CORE;BLUGE;BnLM;corpus;evaluation benchmark;pretrained models},
  doi={10.1109/ACCESS.2026.3701520}
}