VEXMLM

Vocabulary Expansion for Low-Resource Multilingual Language Modeling.

VEXMLM extends xlm-roberta-base with 30,000 Ge'ez-script subword tokens merged natively into its SentencePiece model, then adapts the expanded model with continued masked-language-model pretraining on Amharic and Tigrinya.

Official implementation: https://github.com/hailaykidu/VEXMLM

Languages

Amharic (am) and Tigrinya (ti) — the two highest-resource Ge'ez-script languages. Both are covered by the pretraining corpus and by every reported evaluation.

Model details

Tokenizer

The 30,000 new tokens are merged directly into the SentencePiece model rather than appended as Hugging Face added_tokens. Appending them causes the added-token matcher to run before SentencePiece segmentation, which emits the ▁ word-boundary marker mid-word and corrupts decoding; the merged construction avoids this.

New embedding rows are initialized to the mean of the existing embedding matrix (global_mean), then trained during continued pretraining.

Tokenizer quality

Measured on Amharic and Tigrinya development corpora (2,588 / 2,811 sentences):

Tigrinya fertility falls 45.9% and compression rises 84.9%. Added tokens carry 24.2% (Amharic) and 45.5% (Tigrinya) of token mass, so they are actively used.

Training

Continued MLM pretraining on Amharic and Tigrinya monolingual corpora (200,001 and 200,000 non-empty lines; 9.09M and 6.98M characters). The text comes from a prepared local collection whose original source is not documented; it matches no named corpus available to the authors, its licence is unknown, and it is not redistributed. An earlier attribution to HornMT is ruled out.

Released checkpoint is the best-by-validation-loss model at epoch 56 (24,808 of 26,580 steps): best validation loss 3.7120; the final re-evaluation of this checkpoint gives loss 3.7299 (perplexity 41.67).

Evaluation

Fine-tuned downstream, seeds 42–46, one configuration (hash ce27cc194946) on A100-PCIE-40GB with deterministic kernels. Mean ± standard deviation over 5 seeds.

Supplementary — TiQuAD is a diagnostic task, not a paper benchmark: EM 50.24 ± 0.48, F1 58.90 ± 0.66 (926 questions).

TIGQA has only 67 development questions, too few to support a QA claim on its own; TiQuAD is reported alongside it for that reason.

Ablation — downstream NER OOV accuracy

Tigrinya NER, 4 configurations × 5 seeds. A word is out-of-vocabulary when the baseline xlm-roberta-base tokenizer emits <unk>, fails to round-trip it, or fragments it into more subwords than the expanded tokenizer. All arms are scored on one identical set: 3,491 of 4,677 word types (74.6%).

Vocabulary expansion alone degrades OOV accuracy: the newly added embedding rows are untrained, and the classifier must work around them. Continued pretraining adapts the expanded vocabulary, recovers that loss, and finishes 1.09 points above the baseline.

Note that arms 2 and 3 receive no continued pretraining at all, so the +8.02 attributed to it also includes the effect of 56 additional epochs of training on Amharic/Tigrinya text. Separating embedding adaptation from general continued training would require an unexpanded arm given the same budget, which was not run.

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

tokenizer = AutoTokenizer.from_pretrained("Hailay/VEXMLM")
model = AutoModelForMaskedLM.from_pretrained("Hailay/VEXMLM")
model.eval()

text = "ትግርኛ <mask> ቋንቋ እዩ።"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

mask_pos = (inputs.input_ids[0] == tokenizer.mask_token_id).nonzero()[0, 0]
top = logits[0, mask_pos].topk(3).indices.tolist()
print([tokenizer.decode([t]).strip() for t in top])

This is a masked-language model. For token classification, question answering, or sequence classification, fine-tune it with the corresponding AutoModelFor... class — see the GitHub repository for the fine-tuning scripts and configurations used to produce the results above.

Intended use

Intended for research on Amharic and Tigrinya NLP: as a starting point for fine-tuning on token classification, extractive QA, and sequence classification, and for studying vocabulary expansion in low-resource multilingual models.

Limitations

• Two languages only. Amharic and Tigrinya. Other Ge'ez-script languages were not part of pretraining and are not evaluated here.
• Extractive QA remains weak in absolute terms. TIGQA EM of 2.39 reflects a very small dataset (67 development questions) and a hard task, not a usable QA system.
• Sentiment results are near chance on AfriSenti Amharic (accuracy 0.4978 on a 3-class task).
• Tokenizer parity is not reported. It requires a sentence-aligned parallel corpus, which was unavailable; parity computed on non-parallel text reflects content differences rather than tokenizer fairness.
• The XLM-R baseline is single-seed. Its seed-42 run records are in the repository (results/baselines/xlmr_seed42/); no multi-seed head-to-head claim is made. No downstream Glot500 result is reported.
• Continued pretraining conflates embedding adaptation with additional training budget, as noted in the ablation section.
• The pretraining text is of undocumented origin (licence unknown); samples contain religious translations alongside general web prose, and the model may reflect those distributions and any biases present in them.

Citation

@inproceedings{teklehaymanot2026vexmlm,
  title     = {Vocabulary Expansion for Low-Resource African Languages:
               A Case Study in Amharic and Tigrinya},
  author    = {Teklehaymanot, Hailay Kidu and Yadeta, Debela Desalegn and
               Nejdl, Wolfgang},
  booktitle = {Proceedings of the Workshop on Language Models for
               Underserved Communities (LM4UC) at IJCAI},
  year      = {2026}
}

Accepted at the LM4UC Workshop, IJCAI 2026.

License

Apache 2.0, following xlm-roberta-base.