pl-ModernBERT-512-base

Polish ModernBERT is a family of monolingual ModernBERT-based encoders pretrained for Polish. The family covers two model scales (Base and Large) and two context lengths (512 and 8,192 tokens). The models are general-purpose pretrained encoders intended to serve as strong foundations for a broad range of Polish NLP applications. They can be fine-tuned or adapted for downstream tasks including, but not limited to, text and document classification, sequence labeling, regression, semantic similarity, retrieval, reranking, and representation learning.

📄 Paper: Polish ModernBERT: The Long and Short of Polish Language Understanding
🤗 Model collection: Polish ModernBERT
📚 LongContext benchmark: Dataset

Model family

All variants use the core ModernBERT architecture with RoPE, GeGLU feed-forward layers, pre-normalization, and alternating global/local attention. The Base models use 22 layers with hidden size 768, while the Large models use 28 layers with hidden size 1,024.

Training

The models were pretrained on approximately 44.5B Polish tokens (197 GB after preprocessing) from a curated Polish corpus, Common Crawl (CC-MAIN-2019-43), and the Polish pol_Latn subset of FineTranslations.

Pretraining follows a staged recipe selected through downstream validation: four 512-token stages progressively transition from token-level MLM to whole-word masking, increased emphasis on curated data, and final annealing, followed by long-context continuation to 8,192 tokens. During context extension, the global RoPE theta is increased from 10,000 to 160,000.

Training was performed with Composer in BF16 using StableAdamW and distributed data parallelism across 8 NVIDIA GH200 GPUs.

Evaluation

The models were evaluated on 30 Polish NLU tasks spanning KLEJ, FinBench, the five-task LongContext benchmark, and a range of additional Polish NLP tasks. Each model-task configuration was fine-tuned using five random seeds; scores below are means on a 0–100 scale.

Evaluation Results

The tables below summarize average performance across the main evaluation groups. Scores are reported on a 0–100 scale. Overall Avg. is the macro-average across all 30 individual tasks rather than the average of the four group-level scores.

Base models

Large models

KLEJ

Polish ModernBERT is competitive with the strongest Polish BERT/RoBERTa encoders on KLEJ. The Base variants obtain the highest KLEJ average in both context settings, while the Large variants remain within 0.21 points of the corresponding Polish RoBERTa models.

FinBench

Polish ModernBERT shows particularly strong performance on FinBench. The Base variants achieve the highest FinBench average in both context settings. At Large scale, the 512-token Polish ModernBERT obtains the highest average, while the 8K variant remains close to the corresponding Polish RoBERTa model.

Other Tasks

Across the additional Polish NLP tasks, the Base variants of Polish ModernBERT obtain the highest average in both context settings. At Large scale, the 512-token model achieves the highest average, while the 8K variant remains competitive with the corresponding Polish RoBERTa model.

Long-context performance

Polish ModernBERT shows its largest gains on the LongContext benchmark, which consists of five tasks designed specifically to evaluate long-document understanding. The 8K variants achieve the highest LongContext average at both model scales.

At Base scale, pl-ModernBERT-base improves over pl-RoBERTa-8K-base by 9.68 points (77.15 vs. 67.47) while using 22% fewer parameters (149M vs. 190M). At Large scale, pl-ModernBERT-large improves over the corresponding Polish RoBERTa-8K baseline by 2.61 points (78.49 vs. 75.88).

The 512-token variants are evaluated on LongContext with truncation and should be treated as practical short-context baselines rather than controlled context-length ablations.

Inference efficiency

In a common BF16 inference setup on a single NVIDIA H100, Polish ModernBERT provides favorable quality–efficiency trade-offs relative to the corresponding Polish RoBERTa baselines.

For 512-token inference, the Base and Large models reduce latency by approximately 26% and 47%, respectively, while reducing peak GPU memory usage by 54% and 18%. In the 8K setting, the corresponding latency reductions are 6% and 25%, with peak memory reductions of 24% and 21%.

Usage

from transformers import AutoTokenizer, AutoModel

model_id = "OPI-PIB/pl-ModernBERT-512-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)

text = "W białodrzewiu jaśnie dźni słoneczno, miodzie złoci białopałem żyśnie, drzewia pełni pszczelą i pasieczną, a przez liście kraśnie pęk słowiśnie."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

outputs = model(**inputs)
token_embeddings = outputs.last_hidden_state

For downstream tasks such as classification, regression, sequence labeling, retrieval, or reranking, the pretrained encoder can be adapted to the target task.

The raw checkpoints are not retrieval-specific sentence-embedding models. PIRB retrieval results reported in the paper were obtained after contrastive fine-tuning.

For longer inputs, use one of the corresponding 8K checkpoints from the Polish ModernBERT family.

Flash Attention 2

🏎️ For the highest training and inference efficiency, we recommend using Polish ModernBERT with Flash Attention 2 when supported by your GPU and environment.

pip install flash-attn --no-build-isolation

The model can then be loaded with:

from transformers import AutoModel

model = AutoModel.from_pretrained(
    model_id,
    attn_implementation="flash_attention_2",
    torch_dtype="auto",
)

Authors

Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec

AI Lab, National Information Processing Institute
(Ośrodek Przetwarzania Informacji – Państwowy Instytut Badawczy, OPI PIB)
Warsaw, Poland

Corresponding author: mperelkiewicz@opi.org.pl

Acknowledgments

This work was supported by the Gaia AI Factory project, funded by the European Union under Grant Agreement No. 101314359 through the EuroHPC Joint Undertaking (EuroHPC JU).

We gratefully acknowledge the Polish high-performance computing infrastructure PLGrid (HPC Center: ACK Cyfronet AGH) for providing computational resources and support within computational grant PLG/2025/018315.

Citation

If you use Polish ModernBERT in your work, please cite:

@misc{perełkiewicz2026polishmodernbertlongshort,
      title={Polish ModernBERT: The Long and Short of Polish Language Understanding}, 
      author={Michał Perełkiewicz and Sławomir Dadas and Rafał Poświata and Małgorzata Grębowiec},
      year={2026},
      eprint={2609.01379},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.01379}, 
}