BERTomelo

BERTomelo is a variant of BERT encoders built upon the ModernBERT architecture. It is pretrained from scratch on the Classified Common Crawl Corpus for Portuguese (ClassiCC PT), specifically tailored and specialized for the Portuguese language. BERTomelo try to fill the gap of older Brazilian BERT models with the native long context capability, making it ideal for tasks that require processing extensive documents, mirroring the efficiency of the ModernBERT reference architecture.

Architectural Improvements

BERTomelo leverages several recent advancements in transformer design integrated into the ModernBERT architecture, such as:

• Rotary Positional Embeddings (RoPE): Facilitates robust long context support by encoding positional information through rotation, allowing for better extrapolation to larger sequence lengths.
• Local Global Alternating Attention: Optimizes computational efficiency by alternating between local attention (focusing on neighboring tokens) and global attention (capturing dependencies across the entire sequence).
• Unpadding and FlashAttention 2: Enhances hardware level efficiency by removing unnecessary padding tokens and utilizing an IO aware attention implementation to maximize token throughput.

The pretraining of BERTomelo encompasses 386.48 billion tokens. The custom tokenizer is sourced from ModBERTBr, trained using a Unigram algorithm on the BrWaC corpus and the Wikipedia Portuguese dataset.

Available Models

Usage

You can use BERTomelo leveraging the transformers library since version v4.48.0:

pip install -U transformers>=4.48.0

Since BERTomelo is pretrained using a Masked Language Model (MLM) objective, it can be used directly for inference via AutoModelForMaskedLM or the fill-mask pipeline. It also serves as a robust backbone for fine tuning on downstream tasks such as classification, textual similarity, retrieval, or question answering.

⚠️ Note: We recommend using FlashAttention 2 to achieve the highest processing efficiency and throughput.

Using AutoModelForMaskedLM:

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "unb-labia/BERTomelo-ModernBERT-Base-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
text = "O cachorro mais famoso do brasil é o vira [MASK]."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
predicted_token = tokenizer.decode(predicted_token_id)
print("Predicted token:", predicted_token)

Using a pipeline:

import torch
from transformers import pipeline
from pprint import pprint
pipe = pipeline(
    "fill-mask",
    model="unb-labia/BERTomelo-ModernBERT-Base-v1",
)
input_text = "O Brasil está presente na [MASK] do sul."
results = pipe(input_text)

for result in results:
    print(f"Token: {result['token_str']} | Score: {result['score']:.4f}")

Note: Unlike traditional BERT models, the ModernBERT architecture utilized by BERTomelo does not require token type IDs. You can omit this parameter in most downstream tasks.

Evaluation

The following table summarizes the performance of BERTomelo compared to other prominent models for Brazilian Portuguese across Semantic Textual Similarity (STS), Recognizing Textual Entailment (RTE), and Named Entity Recognition (NER).

Our evaluation pipeline is designed to ensure statistical robustness by separating hyperparameter tuning from the final multi-seed evaluation.

• Hyperparameter Optimization (Optuna): We perform a search using Optuna over a single fixed seed. The objective is to maximize the task-specific target metric (F1-Score for NER, Pearson for STS, and Macro-F1 for Classification/RTE) on the validation set. We optimize the Learning Rate, Weight Decay, and Warmup Ratio. To prevent overfitting and save compute, we apply an Early Stopping mechanism with a patience of 3 epochs.
• Multi-Seed Evaluation (Robustness): Once the optimal hyperparameters are identified, we perform a full fine-tuning of the model across 6 different random seeds.
  • For each seed, the task-specific classification head is initialized from scratch (randomly), while the Transformer backbone loads the pretrained weights.
  • We train for up to 20 epochs, keeping the same 3-epoch Early Stopping rule.
  • Due to load_best_model_at_end=True, the model always reverts to the weights that achieved the highest validation score, regardless of when training halted.
  • Finally, we report the mean and standard deviation (std) of the metrics on the test set across all 6 runs.

Aggregated results (Mean ± Standard Deviation) by model and benchmark.

Limitations

• The evaluation process lacks the availability of standardized long context benchmarks specifically designed for the Portuguese language.
• The current release does not yet incorporate the sequence packing mechanism, which may affect token throughput in specific training scenarios.
• As with all models trained on large scale web data, BERTomelo may reflect biases present in the ClassiCC PT dataset.

Acknowledgments

This work was supported in part by Advanced Micro Devices, Inc. under the AMD AI & HPC Cluster Program. Furthermore, the respective authors are appreciated for providing the ClassiCC-PT, ASSIN2, and LeNER-BR datasets, as well as the developers of ModBERTBr for their foundational work in adapting this architecture for the Brazilian Portuguese language.