XLMRoBERTa for Khmer Language
Training from scratch using Masked Language Modeling task on 5M Khmer sentences or 162M words or 578K unique words for 1M steps.
Training data is created by crawling publicly available publicly news sites and Wikipedia.
Why?
- xlm-roberta-base is big. (279M parameters, while this is only 49M parameters).
- xlm-roberta-base is not optimized for Khmer language.
- xlm-roberta-base Vocab size is bigger (250,002) and this model uses 8000 vocab size.
Usage
from transformers import pipeline
pipe = pipeline("fill-mask", "seanghay/xlm-roberta-khmer-small")
result = pipe("សួស្ដីកម្ពុ<mask>!")
print(result)
[
{"score": 0.8130345344543457, "token": 11, "token_str": "ជា", "sequence": "សួស្ដីកម្ពុជា!"},
{"score": 0.17512884736061096, "token": 160, "token_str": "ជ", "sequence": "សួស្ដីកម្ពុជ!"},
{"score": 0.0034702506382018328, "token": 143, "token_str": "ជា", "sequence": "សួស្ដីកម្ពុ ជា!"},
{"score": 0.00305828545242548, "token": 16, "token_str": "រ", "sequence": "សួស្ដីកម្ពុរ!"},
{"score": 0.0007526700501330197, "token": 133, "token_str": "គ", "sequence": "សួស្ដីកម្ពុគ!"},
]
License
Apache-2.0
Citation
No need. :)