Paper: For more details, please refer to our paper: BERTabaporu: Assessing a Genre-Specific Language Model for Portuguese NLP
Introduction
BERTabaporu is a Brazilian Portuguese BERT model in the Twitter domain. The model has been built from a collection of 238 million tweets written by over 100 thousand unique Twitter users, and conveying over 2.9 billion tokens in total.
Available models
| Model | Arch. | #Layers | #Params |
|---|---|---|---|
pablocosta/bertabaporu-base-uncased | BERT-Base | 12 | 110M |
pablocosta/bertabaporu-large-uncased | BERT-Large | 24 | 335M |
Usage
from transformers import AutoTokenizer # Or BertTokenizer
from transformers import AutoModelForPreTraining # Or BertForPreTraining for loading pretraining heads
from transformers import AutoModel # or BertModel, for BERT without pretraining heads
model = AutoModelForPreTraining.from_pretrained('pablocosta/bertabaporu-base-uncased')
tokenizer = AutoTokenizer.from_pretrained('pablocosta/bertabaporu-base-uncased')
Cite us
@inproceedings{costa-etal-2023-bertabaporu, title = "{BERT}abaporu: Assessing a Genre-Specific Language Model for {P}ortuguese {NLP}", author = "Costa, Pablo Botton and Pavan, Matheus Camasmie and Santos, Wesley Ramos and Silva, Samuel Caetano and Paraboni, Ivandr{'e}", editor = "Mitkov, Ruslan and Angelova, Galia", booktitle = "Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing", month = sep, year = "2023", address = "Varna, Bulgaria", publisher = "INCOMA Ltd., Shoumen, Bulgaria", url = "https://aclanthology.org/2023.ranlp-1.24", pages = "217--223", abstract = "Transformer-based language models such as Bidirectional Encoder Representations from Transformers (BERT) are now mainstream in the NLP field, but extensions to languages other than English, to new domains and/or to more specific text genres are still in demand. In this paper we introduced BERTabaporu, a BERT language model that has been pre-trained on Twitter data in the Brazilian Portuguese language. The model is shown to outperform the best-known general-purpose model for this language in three Twitter-related NLP tasks, making a potentially useful resource for Portuguese NLP in general.", }