Model card for RoBERT-large

language:

• ro

RoBERT-large

Pretrained BERT model for Romanian

Pretrained model on Romanian language using a masked language modeling (MLM) and next sentence prediction (NSP) objective. It was introduced in this paper. Three BERT models were released: RoBERT-small, RoBERT-base and RoBERT-large, all versions uncased.

All models are available:

• RoBERT-small
• RoBERT-base
• RoBERT-large

How to use

# tensorflow
from transformers import AutoModel, AutoTokenizer, TFAutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/RoBERT-large")
model = TFAutoModel.from_pretrained("readerbench/RoBERT-large")
inputs = tokenizer("exemplu de propoziție", return_tensors="tf")
outputs = model(inputs)

# pytorch
from transformers import AutoModel, AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/RoBERT-large")
model = AutoModel.from_pretrained("readerbench/RoBERT-large")
inputs = tokenizer("exemplu de propoziție", return_tensors="pt")
outputs = model(**inputs)

Training data

The model is trained on the following compilation of corpora. Note that we present the statistics after the cleaning process.

Downstream performance

Sentiment analysis

We report Macro-averaged F1 score (in %)

Moldavian vs. Romanian Dialect and Cross-dialect Topic identification

We report results on VarDial 2019 Moldavian vs. Romanian Cross-dialect Topic identification Challenge, as Macro-averaged F1 score (in %).

Diacritics Restoration

Challenge can be found here. We report results on the official test set, as accuracies in %.

BibTeX entry and citation info

@inproceedings{masala2020robert,
  title={RoBERT--A Romanian BERT Model},
  author={Masala, Mihai and Ruseti, Stefan and Dascalu, Mihai},
  booktitle={Proceedings of the 28th International Conference on Computational Linguistics},
  pages={6626--6637},
  year={2020}
}