mRoBERTa Model Card

mRoBERTa is a new multilingual foundational model based on the RoBERTa architecture. It has been pretrained from scratch using 35 European languages and code. The pretraining corpus consists of 12.8TB of high-quality data. This is significantly larger compared to previous state-of-the-art encoder-only foundational models like XLM-RoBERTa-base and XLM-RoBERTa-large, whose training datasets included less multilingual data, amounting to 2.5TB.

Technical Description

Technical details of the mRoBERTa model.

Training Hyperparemeters

How to use

You can use the pipeline for masked language modeling:

>>> from transformers import pipeline
>>> from pprint import pprint
>>> unmasker = pipeline('fill-mask', model='BSC-LT/mRoBERTa')

>>> pprint(unmasker("I love the<mask>of Barcelona.",top_k=3))
[{'score': 0.44038116931915283,
  'sequence': 'I love the city of Barcelona.',
  'token': 31489,
  'token_str': 'city'},
 {'score': 0.10049665719270706,
  'sequence': 'I love the City of Barcelona.',
  'token': 13613,
  'token_str': 'City'},
 {'score': 0.09289316833019257,
  'sequence': 'I love the streets of Barcelona.',
  'token': 178738,
  'token_str': 'streets'}]
>>> pprint(unmasker("Me encanta la<mask>de Barcelona.",top_k=3))
[{'score': 0.17127428948879242,
  'sequence': 'Me encanta la historia de Barcelona.',
  'token': 10559,
  'token_str': 'historia'},
 {'score': 0.14173351228237152,
  'sequence': 'Me encanta la ciudad de Barcelona.',
  'token': 19587,
  'token_str': 'ciudad'},
 {'score': 0.06284074485301971,
  'sequence': 'Me encanta la vida de Barcelona.',
  'token': 5019,
  'token_str': 'vida'}]
>>> pprint(unmasker("M'encanta la<mask>de Barcelona.",top_k=3))
[{'score': 0.35796159505844116,
  'sequence': "M'encanta la ciutat de Barcelona.",
  'token': 17128,
  'token_str': 'ciutat'},
 {'score': 0.10453521460294724,
  'sequence': "M'encanta la història de Barcelona.",
  'token': 35763,
  'token_str': 'història'},
 {'score': 0.07609806954860687,
  'sequence': "M'encanta la gent de Barcelona.",
  'token': 15151,
  'token_str': 'gent'}]

Alternatively, you can also extract the logits associated with the sequences and perform the calculations by hand:

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

model = AutoModelForMaskedLM.from_pretrained("BSC-LT/mRoBERTa")
tokenizer = AutoTokenizer.from_pretrained("BSC-LT/mRoBERTa")

outputs = model(**tokenizer("The capital of Spain is<mask>", return_tensors="pt")).logits

# The index of "<mask>" token is -2 given that the -1 position is the EOS token "</s>".
predicted_token = tokenizer.decode(torch.argmax(outputs[0,-2,:])) 

print(f"The decoded element is \"{predicted_token}\"." ) # This will give "Madrid"

In most of the evaluations presented below, the model is adjusted to each use case using specific logits to encode the text.

Data

Pretraining Corpus

The training corpus consists of 35 European languages and 92 programming languages, amounting to a total of 12.8TB of high-quality data.

This highly multilingual corpus is predominantly composed of data from Colossal OSCAR, which contributes a significant 66.06% of the total tokens. Following this, Starcoder provides 11.91%, and Spanish Crawling adds 3.34%. The next largest sources are French PD at 3.12% and Proof Pile at 1.98%. Other notable contributions include Macocu, Pile of Law, and Eurlex, each contributing around 1.5% to 1.3%. These major sources collectively form the bulk of the corpus, ensuring a rich and diverse dataset for training the language model. The remaining 10% comes from smaller sources in various languages.

The final pretraining language distribution split by language can be seen in the following picture:

Further details about the pretraining corpus can be found here, as it is the same as the Salamandra foundational model.

Multilingual Evaluation and Performance

Evaluation is done using multilingual benchmarks in order to assess the multilingual capabilities of the models.

The following multilingual benchmarks have been considered:

The following base foundational models have been considered:

RESULTS

This section presents results across various multilingual benchmarks, with the maximum values highlighted in bold and the second-highest values underlined.

XTREME Benchmark

The Cross-lingual TRansfer Evaluation of Multilingual Encoders (XTREME) benchmark is a benchmark for the evaluation of the cross-lingual generalization ability of pre-trained multilingual models. It includes nine tasks that collectively require reasoning about different levels of syntax and semantics. The languages in XTREME are selected to maximize language diversity, coverage in existing tasks, availability of training data, and overlap with the languages present during pre-training of the models.

🔵 Sentence Classification

🔵 XNLI

Metric used: Accuracy.

🔵 PAWS-X

Metric used: Accuracy.

🟣 Structured Prediction: POS

🟣 POS (UDPOS)

Metric used: F1.

🟣 NER (PANX)

Metric used: F1.

⚪️ Sentence Retrieval

⚪️ BUCC2018

Metric used: F1.

⚪️ Tatoeba

Metric used: Accuracy.

⚫ Question Answering

⚫ XQUAD

Metric used: F1.

⚫ MLQA

Metric used: F1.

⚫ TyDiQA

Metric used: F1.

CLUB Benchmark

The Catalan Language Understanding Benchmark consists of 6 tasks: Named Entity Recognition (NER), Part-of-Speech Tagging (POS), Semantic Textual Similarity (STS), Text Classification (TC), Textual Entailment (TE), and Question Answering (QA). This benchmark evaluates the model's capabilities in the Catalan language.

This comparison also includes RoBERTa-ca, a model derived from mRoBERTa by applying vocabulary adaptation and performing continual pre-training on a 95GB Catalan-only corpus. For further details, visit here.

Galician Benchmark

To evaluate performance in Galician, the models are tested on two tasks highlighted in Bertinho's paper:

• NER task: NER task using the SLI NERC dataset.
• POS task: POS task using the Universal Dependencies dataset.

BasqueGLUE Benchmark

To assess the model performance in the Basque language, the BasqueGLUE Benchmark is used as the baseline. BasqueGLUE has been elaborated from previously existing datasets and following similar criteria to those used for the construction of GLUE and SuperGLUE. Some of the tasks has been slightly adapted to easily assess all the models (e.g, FMTODeu_slot is originally described as "Slot filling" task, but it is evaluated as NERC, since it follows the BIO annotation scheme).

Additional information

Author

The Language Technologies Lab from Barcelona Supercomputing Center.

Contact

For further information, please send an email to langtech@bsc.es.

Copyright

Copyright(c) 2025 by Language Technologies Lab, Barcelona Supercomputing Center.

Funding

This work has been promoted and financed by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia - Funded by EU – NextGenerationEU within the framework of the project Modelos del Lenguaje.

Acknowledgements

This project has benefited from the contributions of numerous teams and institutions through data contributions.

In Catalonia, many institutions have been involved in the project. Our thanks to Òmnium Cultural, Parlament de Catalunya, Institut d'Estudis Aranesos, Racó Català, Vilaweb, ACN, Nació Digital, El món and Aquí Berguedà.

At national level, we are especially grateful to our ILENIA project partners: CENID, HiTZ and CiTIUS for their participation. We also extend our genuine gratitude to the Spanish Senate and Congress, Fundación Dialnet, Fundación Elcano and the ‘Instituto Universitario de Sistemas Inteligentes y Aplicaciones Numéricas en Ingeniería (SIANI)’ of the University of Las Palmas de Gran Canaria.

At the international level, we thank the Welsh government, DFKI, Occiglot project, especially Malte Ostendorff, and The Common Crawl Foundation, especially Pedro Ortiz, for their collaboration.

Their valuable efforts have been instrumental in the development of this work.

Disclaimer

Be aware that the model may contain biases or other unintended distortions. When third parties deploy systems or provide services based on this model, or use the model themselves, they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations, including those governing the use of Artificial Intelligence.

The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.

License

Apache License, Version 2.0