PortBERT: Navigating the Depths of Portuguese Language Models
PortBERT is a family of RoBERTa-based language models pre-trained from scratch on the Portuguese portion of OSCAR23 and MC4 (deduplicated variants of CulturaX). The models are designed to offer strong downstream performance in Portuguese NLP tasks, while providing insights into the cost-performance tradeoffs of training across hardware backends.
We release two variants:
PortBERT-base: 126M parameters, trained on 8× A40 GPUs (fp32)PortBERT-large: 357M parameters, trained on TPUv4-128 pod (fp32)
Model Details
| Detail | PortBERT-base | PortBERT-large |
|---|---|---|
| Architecture | RoBERTa-base | RoBERTa-large |
| Parameters | ~126M | ~357M |
| Tokenizer | GPT-2 style (52k vocab) | Same |
| Pretraining corpus | deduplicated mC4 and OSCAR 23 from CulturaX | Same |
| Objective | Masked Language Modeling | Same |
| Training time | ~27 days on 8× A40 | ~6.2 days on TPUv4-128 pod |
| Precision | fp32 | fp32 |
| Framework | fairseq | fairseq |
Downstream Evaluation (ExtraGLUE)
We evaluate PortBERT on ExtraGLUE, a Portuguese adaptation of the GLUE benchmark. Fine-tuning was conducted using HuggingFace Transformers, with NNI-based grid search over batch size and learning rate (28 configurations per task). Each task was fine-tuned for up to 10 epochs. Metrics were computed on validation sets due to the lack of held-out test sets.
AVG score averages the following metrics:
- STSB Spearman
- STSB Pearson
- RTE Accuracy
- WNLI Accuracy
- MRPC Accuracy
- MRPC F1
🧪 Evaluation Results
Legend: Bold = best, italic = second-best per model size.
| Model | STSB_Sp | STSB_Pe | STSB_Mean | RTE_Acc | WNLI_Acc | MRPC_Acc | MRPC_F1 | AVG |
|---|---|---|---|---|---|---|---|---|
| Large models | ||||||||
| XLM-RoBERTa_large | 90.00 | 90.27 | 90.14 | 82.31 | 57.75 | 90.44 | 93.31 | 84.01 |
| EuroBERT-610m | 88.46 | 88.59 | 88.52 | 78.34 | 59.15 | 91.91 | 94.20 | 83.44 |
| PortBERT_large | 88.53 | 88.68 | 88.60 | 72.56 | 61.97 | 89.46 | 92.39 | 82.26 |
| BERTimbau_large | 89.40 | 89.61 | 89.50 | 75.45 | 59.15 | 88.24 | 91.55 | 82.23 |
| Base models | ||||||||
| RoBERTaLexPT_base | 86.68 | 86.86 | 86.77 | 69.31 | 59.15 | 89.46 | 92.34 | 80.63 |
| PortBERT_base | 87.39 | 87.65 | 87.52 | 68.95 | 60.56 | 87.75 | 91.13 | 80.57 |
| RoBERTaCrawlPT_base | 87.34 | 87.45 | 87.39 | 72.56 | 56.34 | 87.99 | 91.20 | 80.48 |
| BERTimbau_base | 88.39 | 88.60 | 88.50 | 70.40 | 56.34 | 87.25 | 90.97 | 80.32 |
| XLM-RoBERTa_base | 85.75 | 86.09 | 85.92 | 68.23 | 60.56 | 87.75 | 91.32 | 79.95 |
| EuroBERT-210m | 86.54 | 86.62 | 86.58 | 65.70 | 57.75 | 87.25 | 91.00 | 79.14 |
| AlBERTina 100M PTPT | 86.52 | 86.51 | 86.52 | 70.04 | 56.34 | 85.05 | 89.57 | 79.01 |
| AlBERTina 100M PTBR | 85.97 | 85.99 | 85.98 | 68.59 | 56.34 | 85.78 | 89.82 | 78.75 |
| AiBERTa | 83.56 | 83.73 | 83.65 | 64.98 | 56.34 | 82.11 | 86.99 | 76.29 |
| roBERTa PT | 48.06 | 48.51 | 48.29 | 56.68 | 59.15 | 72.06 | 81.79 | 61.04 |
Fairseq Checkpoint
Get the fairseq checkpoint here.
Citations
If you use PortBERT in your research, please cite the following paper:
@book{scheible-schmitt-etal-2025-portbert,
author = {\textbf{Scheible-Schmitt}, \textbf{Raphael} and He, Henry and Mendes, Armando B.},
title = {PortBERT: Navigating the Depths of Portuguese Language Models},
booktitle = {Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models},
month = {September},
year = {2025},
address = {Varna, Bulgaria},
publisher = {INCOMA Ltd., Shoumen, BULGARIA},
pages = {59--71},
abstract = {Transformer models dominate modern NLP, but efficient, language-specific models remain scarce. In Portuguese, most focus on scale or accuracy, often neglecting training and deployment efficiency. In the present work, we introduce PortBERT, a family of RoBERTa-based language models for Portuguese, designed to balance performance and efficiency. Trained from scratch on over 450 GB of deduplicated and filtered mC4 and OSCAR23 from CulturaX using fairseq, PortBERT leverages byte-level BPE tokenization and stable pre-training routines across both GPU and TPU processors. We release two variants, PortBERT base and PortBERT large, and evaluate them on ExtraGLUE, a suite of translated GLUE and SuperGLUE tasks. Both models perform competitively, matching or surpassing existing monolingual and multilingual models. Beyond accuracy, we report training and inference times as well as fine-tuning throughput, providing practical insights into model efficiency. PortBERT thus complements prior work by addressing the underexplored dimension of compute-performance tradeoffs in Portuguese NLP. We release all models on Huggingface and provide fairseq checkpoints to support further research and applications.},
url = {https://aclanthology.org/2025.globalnlp-1.8},
doi = {https://doi.org/10.26615/978-954-452-105-9-008}
}
📜 License
MIT License