XLM-RoBERTa Large Fine-tuned for Sinhala Spell Correction
An XLM-RoBERTa-large model fine-tuned for Sinhala spell correction as part of the LMSpell project.
- Model: XLM-RoBERTa-large
- Base model:
FacebookAI/xlm-roberta-large - Language: Sinhala (
si) - Task: Sinhala Spell Correction (SSC)
- Dataset: Sinhala Spell Correction Dataset
- License: CC BY 4.0
- Project: LMSpell
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_name = "lm-spell/xlm-roberta-large-ft-ssc"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
ZWJ_token = "<ZWJ>"
ZWJ_unicode = "\u200D"
text = "යින් හැඟෙන්නේ පවත්නා අධ්යාපන ක්රමයේ පවතින"
text = text.replace(ZWJ_unicode, ZWJ_token)
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
predicted_ids = outputs.logits.argmax(dim=-1)
decoded = tokenizer.decode(
predicted_ids[0],
skip_special_tokens=True
)
decoded = decoded.replace(ZWJ_token, ZWJ_unicode)
print(decoded)
Zero Width Joiner Handling
XLM-RoBERTa family of models do not preserve the Unicode Zero Width Joiner (\u200D) during tokenization and generation. To address this, LMSpell represents the Zero Width Joiner using a custom special token, <ZWJ>, during inference.
Before tokenization, occurrences of \u200D are replaced with <ZWJ>. After generation and decoding, <ZWJ> is converted back to the original Unicode Zero Width Joiner (\u200D).
This ensures that Zero Width Joiners are preserved throughout the spell-correction process.
Limitations
- The model is specifically fine-tuned for Sinhala spell correction and may not generalize to other languages.
- Performance may vary for informal text, social media content, code-mixed text, and text containing uncommon or domain-specific vocabulary.
- The model may occasionally modify text that is already correctly spelled.
- XLM-RoBERTa is an encoder-only masked language model and does not generate text autoregressively.
Research
For details on the training methodology, dataset preparation, evaluation, results, limitations, and comparisons with other models, please refer to the LMSpell paper:
LMSpell: Spell Correction with Pre-Trained Language Models
A. Gunathilake, N. Karunarathna, T. Bandaranayake, S. Ranathunga, N. de Silva and N. Jayatilleke, "LMSpell: Spell Correction with Pre-Trained Language Models," 2026 Moratuwa Engineering Research Conference (MERCon), Moratuwa, Sri Lanka, 2026, pp. 503-508.
- Paper: https://doi.org/10.1109/MERCon71835.2026.11691371
- Extended Preprint: https://arxiv.org/html/2512.05414v3
Citation
@INPROCEEDINGS{11691371,
author={Gunathilake, Akesh and Karunarathna, Nadil and Bandaranayake, Tharusha and Ranathunga, Surangika and de Silva, Nisansa and Jayatilleke, Nevidu},
booktitle={2026 Moratuwa Engineering Research Conference (MERCon)},
title={LMSpell: Spell Correction with Pre-Trained Language Models},
year={2026},
pages={503-508},
doi={10.1109/MERCon71835.2026.11691371}
}