GeistBERT

GeistBERT is a German language model trained on a for the most part deduplicated corpus including OSCAR23, OPUS, and MC4. It builds on GottBERT while introducing Whole Word Masking (WWM) to improve contextual language representation. Achieving state-of-the-art among base models, the model also performs competitively with larger ones on several German NLP benchmarks.

Training Data

GeistBERT was trained on a diverse German corpus combining:

• OSCAR23, OPUS, and MC4 (for the most part deduplicated)
• German Wikipedia
• OpenLegalData
• Europarl, EUbookshop, ECB, and EuroPat
• OpenSubtitles and TildeMODEL

The dataset amounts to approximately 1.3T tokens, shuffled for improved variance.

Training Procedure

Hardware

• Training was conducted on multiple GPUs, including NVIDIA RTX 3090 (24GB VRAM).

Hyperparameters

Performance

GeistBERT achieves SOTA results on multiple tasks:

• NER: CoNLL 2003, GermEval 2014
• Text Classification: GermEval 2018 (coarse & fine), 10kGNAD
• NLI: German subset of XNLI

Mertics:

• NER and Text Classification: F1 Score
• NLI: Accuracy

Details:

• bold values indicate the best performing model within one architecure (base, large), undescored values the second best.

Intended Use

This model is designed for German NLP tasks, including:

• Text classification
• Named Entity Recognition (NER)
• Machine Translation Pre-training
• Document Understanding

Limitations

• Trained on unfiltered data, meaning some redundant or lower-quality samples may be present.
• While deduplication was applied to specific subcorpora, the full corpus was not manually curated.

Fairseq Checkpoint

Get the fairseq checkpoint here.

Citations

If you use GeistBERT in your research, please cite the following paper:

@book{scheible-schmitt-frei-2025-geistbert,
  author    = {\textbf{Scheible-Schmitt}, \textbf{Raphael}  and  Frei, Johann},
  title     = {GeistBERT: Breathing Life into German NLP},
  booktitle      = {Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models},
  month          = {September},
  year           = {2025},
  address        = {Varna, Bulgaria},
  publisher      = {INCOMA Ltd., Shoumen, BULGARIA},
  pages     = {42--50},
  abstract  = {Advances in transformer-based language models have highlighted the benefits of language-specific pre-training on high-quality corpora. In this context, German NLP stands to gain from updated architectures and modern datasets tailored to the linguistic characteristics of the German language. GeistBERT seeks to improve German language processing by incrementally training on a diverse corpus and optimizing model performance across various NLP tasks. We pre-trained GeistBERT using fairseq, following the RoBERTa base configuration with Whole Word Masking (WWM), and initialized from GottBERT weights. The model was trained on a 1.3 TB German corpus with dynamic masking and a fixed sequence length of 512 tokens. For evaluation, we fine-tuned the model on standard downstream tasks, including NER (CoNLL 2003, GermEval 2014), text classification (GermEval 2018 coarse/fine, 10kGNAD), and NLI (German XNLI), using $F_1$ score and accuracy as evaluation metrics. GeistBERT achieved strong results across all tasks, leading among base models and setting a new state-of-the-art (SOTA) in GermEval 2018 fine text classification. It also outperformed several larger models, particularly in classification benchmarks. To support research in German NLP, we release GeistBERT under the MIT license.},
  url       = {https://aclanthology.org/2025.globalnlp-1.6},
  doi       = {https://doi.org/10.26615/978-954-452-105-9-006}
}