Mari mBERT: Fine-tuning Multilingual BERT for the Mari Language
This repository contains the code and methodology for fine-tuning bert-base-multilingual-cased (mBERT) on a monolingual Mari language corpus using Masked Language Modeling (MLM). The project was developed as part of a Master's thesis in Linguistics at Volga State University of Technology (VSTU), within the research area of AI-assisted language revitalization.
Background
The Mari language is a Finno-Ugric language of the Volga region classified as low-resource: despite an active speaker community, it remains underrepresented in digital NLP infrastructure. This project addresses that gap by adapting a multilingual transformer model to Mari through targeted fine-tuning, following a methodology comparable to the LakotaBERT project.
The central technical contribution is the extension of the mBERT tokenizer vocabulary with Mari-specific graphemes absent from the standard token set, combined with MLM fine-tuning on a large monolingual corpus. The result is a reproducible, open-source baseline model for Mari NLP tasks.
Dataset
- Corpus:
mari-lab/mari-monolingual-corpus(Mari Laboratory, Hugging Face) - Size: approximately 1.4 million lines of Mari text
- Split: 90% training / 10% validation
- Format: one sentence per line (JSONL)
Model
- Base:
google-bert/bert-base-multilingual-cased - Task: Masked Language Modeling (MLM), 15% masking probability
- Tokenizer extension: 13 Mari graphemes added (
ҥ ӧ ӱ ӹ ӓ ӆ ӊand uppercase variants) - Hardware: NVIDIA A100 80 GB
- Precision: bfloat16 (native A100 Tensor Core support)
Training Configuration
| Parameter | Value |
|---|---|
| Sequence length | 200 tokens |
| Batch size | 128 |
| Gradient accumulation | 1 |
| Epochs | 5 |
| Learning rate | 3e-5 |
| MLM probability | 15% |
| Optimizer | AdamW (fused) |
| Weight decay | 0.01 |
| Warmup ratio | 0.06 |
Evaluation Results
Evaluation was performed on 2,000 sampled sequences (13,292 masked positions), following the LakotaBERT Table 4 protocol.
| Metric | Mari mBERT (this work) | LakotaBERT | mBERT baseline |
|---|---|---|---|
| Accuracy | 53.57% | 51.48% | 54.42% |
| Precision | 0.54 | 0.56 | 0.50 |
| F1-Score | 0.54 | 0.49 | 0.54 |
| MRR | 0.63 | 0.51 | 0.51 |
| CER | 0.32 | 0.43 | 0.38 |
| Hit@10 | 0.81 | 0.31 | 0.24 |
| BLEU | 0.41 | 0.09 | 0.09 |
The model shows strong ranking performance (MRR, Hit@10) relative to comparable low-resource projects, attributable in part to the typological proximity of Mari to other Finno-Ugric languages already represented in the mBERT pretraining corpus (Finnish, Estonian, Hungarian).
Output Artifacts
After training, the following files are saved to the output directory:
model.safetensors— fine-tuned model weightstokenizer_config.json— extended tokenizer configurationmetrics.json— full evaluation metricstraining_dynamics.png— train/validation loss curvesmetrics_summary.png— bar chart of evaluation metrics
Requirements
torch >= 2.0
transformers
datasets
nltk
numpy
pandas
matplotlib
tqdm
Install dependencies:
pip install torch transformers datasets nltk numpy pandas matplotlib tqdm
Usage
Prepare the corpus as a JSONL file with one field text per line, then run:
python mari_mbert_a100.py
Key environment variables for configuration:
| Variable | Default | Description |
|---|---|---|
SEQ | 200 | Sequence length |
BS | 128 | Batch size per device |
EP | 5 | Number of training epochs |
LR | 3e-5 | Learning rate |
WORKERS | 16 | DataLoader worker count |
METRICS_SAMPLE | 2000 | Sequences sampled for evaluation |
USE_COMPILE | 0 | Enable torch.compile (experimental) |
Intended Applications
The fine-tuned model is intended as a foundation for downstream Mari NLP tools, including spell-checking and autocorrection systems, predictive text input for mobile devices and messengers, fill-in-the-blank and grammar exercises for language learners, and further fine-tuning for POS tagging and syntactic parsing.
Limitations
- The training corpus is monolingual; parallel Mari-Russian data was not used
- Dialectal variation between Hill Mari and Meadow Mari is not separated in the corpus
- Automated metrics do not substitute for evaluation by native speakers
Reproducibility
All code is published under an open license. The training corpus is publicly available on Hugging Face. Any researcher can reproduce the full experiment using the provided script and corpus.
Citation
If you use this work, please cite:
Zadvornykh, A. S. Use of AI systems to preserve and revitalize languages. Yoshkar-Ola, 24 June 2026. Yoshkar-Ola: VSTU, 2026.
Lexei1297/mari_mbert
Acknowledgments
- Mari Laboratory for the open monolingual Mari corpus
- Google Research for the
bert-base-multilingual-casedmodel - The LakotaBERT project for the evaluation methodology