AfroXLMR-base-Wolof
AfroXLMR-base adapted to Wolof through continued masked language modeling (MLM). AfroXLMR covers 17 African languages; Wolof is not one of them.
Training
| Base model | Davlan/afro-xlmr-base |
| Objective | MLM, masking probability 0.15 |
| Data | GalsenAI CLM data, galsenai/cleaned_data_for_clm (train split, 958,646 texts), after decontamination |
| Sequence length | 512 tokens (concatenated, chunked) |
| Effective batch size | 64 sequences |
| Epochs | 3 |
| Learning rate | 5e-5, linear schedule, 5% warmup, weight decay 0.01 |
| Hardware | 1 × NVIDIA H200 |
Data. galsenai/cleaned_data_for_clm is derived from the
Wolof Centralized Corpus:
code blocks removed inside lines, NFC normalization, random split (seed 42),
and removal of evaluation texts with a near-duplicate in the training split.
Decontamination. Every training document sharing at least one word 8-gram with
MasakhaNER 2.0 Wolof (train/dev/test) or SIB-200 wol_Latn (all splits) was removed:
1.1% of documents, 0.7% of words. After filtering, no evaluation sentence is found
in the training data.
Evaluation
Mean ± standard deviation over 3 fine-tuning seeds; model selection on the dev/validation set.
| Model | MLM perplexity ↓ | NER F1 (MasakhaNER 2.0, wol) ↑ | SIB-200 accuracy (wol_Latn) ↑ | SIB-200 macro-F1 ↑ |
|---|---|---|---|---|
| AfroXLMR-base | 38.72 | 84.35 ± 0.57 | 62.75 ± 2.44 | 59.38 ± 1.51 |
| AfroXLMR-base-Wolof (this model) | 2.32 | 87.29 ± 0.18 | 76.80 ± 0.23 | 74.13 ± 0.40 |
- MLM perplexity: test split of
galsenai/cleaned_data_for_clm(4,282 texts), fixed masking (same tokenizer for both models). - NER: entity-level F1 (seqeval). SIB-200: 7-class topic classification.
Usage
from transformers import pipeline
fill = pipeline("fill-mask", model="m-a-d-i/afroxlmr-base-wolof")
fill("Dakar mooy <mask> bu Senegaal.")
Limitations
- Evaluated on two tasks only (NER, topic classification); the SIB-200 test set has 204 examples.
- Performance on other Wolof varieties, orthographies or domains is not measured.
References
- XLM-R: Conneau et al., 2020. arXiv:1911.02116
- AfroXLMR: Alabi et al., COLING 2022. arXiv:2204.06487
- MasakhaNER 2.0: Adelani et al., EMNLP 2022. arXiv:2210.12391
- SIB-200: Adelani et al., EACL 2024. arXiv:2309.07445