bslm-mlm-25M-ptdrw
A product-domain masked language model fully fine-tuned from BAAI/bge-m3-retromae on Basalam product titles and descriptions. The text is mostly Persian, and the model keeps the original XLM-RoBERTa-compatible tokenizer and <mask> token.
Source repository
- Project repository: https://git.basalam.dev/mj.alizade/mlm
- Training script: https://git.basalam.dev/mj.alizade/mlm/-/blob/bf2f0253239ba0597e4bf64fe1550d5553a6c8d2/scripts/train_full_finetune_mlm.py
- Training run output:
outputs/full_finetune_bf16/model
Model details
- Base checkpoint:
BAAI/bge-m3-retromae - Architecture:
XLMRobertaForMaskedLM - Adaptation method: full-parameter MLM fine-tuning
- Training precision: fp32 master weights with bf16 mixed precision
- Parameters: 568,006,802 trainable / total parameters
- Vocabulary size: 250,002
- Maximum sequence length: 8,192 tokenizer tokens
Training setup
The model was trained with the repository's opt-in full fine-tuning path (scripts/train_full_finetune_mlm.py) across CUDA devices 2,3,5 using streaming product-domain MLM data.
Key recorded settings from the training run:
| Setting | Value |
|---|---|
| Training MLM probability | 0.35 |
| Evaluation MLM probability | 0.15 |
| Block size | 256 |
| Learning rate | 2e-5 |
| LR scheduler | cosine |
| Warmup ratio | 0.03 |
| Weight decay | 0.01 |
| Per-device train batch size | 12 |
| Gradient accumulation steps | 2 |
| Effective train batch | 72 |
| Max steps | 244,039 |
| Gradient checkpointing | enabled |
Evaluation
Final saved model evaluation on the fixed held-out validation JSONL (data/streaming/full/validation.jsonl) with MLM probability 0.15:
| Metric | Value |
|---|---|
| Validation products | 11,097 |
| Validation blocks | 7,671 |
| Eval loss | 0.2777938857 |
| Perplexity | 1.3202140541 |
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline
model_id = "mjaliz/bslm-mlm-25M-ptdrw"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
print(fill_mask("این محصول برای <mask> مناسب است."))
Use <mask> for masked-token examples, not [MASK].
Intended use
This checkpoint is intended for Persian/product-domain masked language modeling experiments and downstream domain adaptation research. It is not a general-purpose generative model.
Data note
Training data came from product titles and descriptions in the Basalam product domain. The underlying dataset is not included in this model repository.