moBERTo

Paper name: This model is referred to as moBERTo-SWM-8k (PT tok.) in the moBERTo paper. It is the best-performing variant of the moBERTo family, achieving the highest average reranking nDCG@10 across three Portuguese retrieval benchmarks and the best PLUE-PT score.

moBERTo is a Portuguese adaptation of ModernBERT, obtained through continued pretraining on a curated 12-billion-token Portuguese corpus (60B training tokens, 5 epochs) followed by a long-context post-training phase at 8,192-token context.

It combines four adaptation strategies:

1. Continued pretraining from the original ModernBERT-base checkpoint, preserving the long-context capabilities learned during the original 2T-token English pretraining.
2. Portuguese tokenizer with vocabulary optimized for Portuguese text.
3. Subword Matching (SWM) embedding transfer, which initializes each new Portuguese token's embedding as a combination of the original ModernBERT subword embeddings, keeping the model close to its pretrained representation space.
4. Long-context post-training at 8,192 tokens for an additional 10B tokens.

The model preserves all architectural advances of ModernBERT: rotary positional embeddings (RoPE), alternating local–global attention, FlashAttention, and unpadding, with a native context window of 8,192 tokens.

Model Details

Quick Start

long_text = "..."  # documento longo em português
inputs = tokenizer(long_text, max_length=8192, truncation=True, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

Recommended for downstream tasks

This model is best used as a backbone for fine-tuning on:

• Cross-encoder reranking (information retrieval)
• Document classification
• Named entity recognition
• Natural language inference / semantic textual similarity
• Long-document retrieval (up to 8,192 tokens)

Evaluation Results

All metrics are reported on Portuguese benchmarks. Best results are in bold; second-best are underlined.

Information Retrieval (Reranking, nDCG@10)

Cross-encoder reranking, fine-tuned on mMARCO-PT triples.

Long-Context Retrieval (MLDR, nDCG@10)

Classification (F1)

• Docs: document type classification (news, legal, academic, etc.)
• Educ.: educational content detection

NLU and NER (F1)

Note on GLUE: As expected from continued pretraining on Portuguese, English performance degrades. ModernBERT-base remains the strongest on GLUE (0.8301);

Key Findings (from the paper's ablations)

1. Continued pretraining > training from scratch. Especially for long-context: moBERTo achieves 0.5777 on MLDR@8192 vs. 0.1405 for a from-scratch baseline trained on the same Portuguese budget. The original 2T-token ModernBERT pretraining provides representations that transfer effectively even when continued pretraining itself uses only 1,024-token sequences.
2. Tokenizer adaptation helps token-level tasks but disrupts long context. Moving to a Portuguese tokenizer improves PLUE-PT and LeNER-Br but hurts MLDR@8192 (drops by ~11 points without embedding transfer).
3. SWM embedding transfer mitigates the long-context degradation. By initializing new Portuguese embeddings as combinations of the original subword embeddings, SWM recovers most of the long-context performance lost by tokenizer adaptation alone.
4. Long-context post-training yields the strongest reranker. moBERTo (this model) achieves the highest average reranking nDCG@10 (0.5255) and the best PLUE-PT score (0.6980).

Training Data

The pretraining corpus was curated from the Portuguese subset of FineWeb-2 and further filtered using the educational and STEM classifiers from ClassiCC-PT. The final corpus comprises ~12 billion tokens, roughly six times larger than BrWaC, covering a broad range of domains and topics in Portuguese.

The training data has been publicly released alongside the model.

Training Procedure

Phase 1 — Continued pretraining (60B tokens at 1,024 context)

Phase 2 — Long-context post-training (10B tokens at 8,192 context)

Same hyperparameters as Phase 1, except:

Related Models in the moBERTo Family

Citation

@misc{laitz2026mobertomodernencoderportuguese,
      title={moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT}, 
      author={Thiago Laitz and Thales Sales Almeida and João Guilherme Alves Santos and Giovana Kerche Bonás},
      year={2026},
      eprint={2606.22722},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.22722}, 
}