RooseBERT-scr-cased

RooseBERT is a domain-specific BERT-based language model pre-trained on English political debates and parliamentary speeches. It is designed to capture the distinctive features of political discourse, including domain-specific terminology, implicit argumentation, and strategic communication patterns.

This variant — scr-cased — was trained from scratch (SCR) with a custom cased WordPiece tokenizer built from the political debate corpus, giving it a vocabulary better suited to political language than standard BERT.

📄 Paper: RooseBERT: A New Deal For Political Language Modelling
💻 GitHub: https://github.com/deborahdore/RooseBERT

Model Details

The SCR approach trains BERT entirely from scratch on domain-specific data, using a custom tokenizer. This allows political terms like deterrent, endorse, bureaucrat, statutorily, and consequential to be represented as single tokens — whereas standard BERT would split them into multiple sub-tokens. The custom vocabulary shares only ~56% of its tokens with bert-base-cased.

Training Data

RooseBERT was pre-trained on 11GB of English political debate transcripts spanning 1919–2025, drawn from:

All datasets were sourced from authoritative, official political settings. Pre-processing removed hyperlinks, markup tags, and collapsed whitespace.

Intended Use

RooseBERT is intended as a base model for fine-tuning on downstream NLP tasks related to political discourse analysis. It is especially well-suited for:

• Sentiment Analysis of parliamentary speeches and debates
• Stance Detection (support/oppose classification)
• Argument Component Detection and Classification (claims and premises)
• Argument Relation Prediction and Classification (support/attack/no-relation)
• Motion Policy Classification
• Named Entity Recognition in political texts

How to Use

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("ddore14/RooseBERT-scr-cased")
model = AutoModelForMaskedLM.from_pretrained("ddore14/RooseBERT-scr-cased")

For fine-tuning on a downstream classification task:

from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer

tokenizer = AutoTokenizer.from_pretrained("ddore14/RooseBERT-scr-cased")
model = AutoModelForSequenceClassification.from_pretrained(
    "ddore14/RooseBERT-scr-cased",
    num_labels=2
)

# Recommended fine-tuning hyperparameters (from paper):
# learning_rate ∈ {2e-5, 3e-5, 5e-5}
# batch_size ∈ {8, 16, 32}
# epochs ∈ {2, 3, 4}

Evaluation Results

RooseBERT was evaluated across 10 datasets covering 6 downstream tasks. Results below are for RooseBERT-scr-cased (Macro F1 unless noted).

RooseBERT outperforms BERT on 9 out of 10 tasks, with improvements ranging from 4% to 15% on classification tasks. NER performance is comparable, as the NEREx dataset uses general (non-political) entity categories. Results are averaged over 5 runs with different random seeds.

Perplexity on held-out political debate data:

Available Variants

SCR (from scratch) models use a custom political vocabulary; CONT (continued pre-training) models are initialised from the original BERT weights and vocabulary.

Limitations

• RooseBERT is trained exclusively on English political debates. Cross-lingual use is not supported.
• The model may reflect biases present in official political speech, including over-representation of certain geopolitical perspectives.
• Performance on NER tasks does not benefit from domain-specific pre-training when entity categories are general rather than politically specific.
• As with all encoder-only models, RooseBERT is best suited to classification and labelling tasks rather than generation.

Citation

If you use RooseBERT in your research, please cite:

@article{dore2025roosebert,
  title={RooseBERT: A New Deal For Political Language Modelling},
  author={Dore, Deborah and Cabrio, Elena and Villata, Serena},
  journal={arXiv preprint arXiv:2508.03250},
  year={2025}
}

Acknowledgements

This work was supported by the French government through the 3IA Côte d'Azur programme (ANR-23-IACL-0001). Computing resources were provided by GENCI at IDRIS (grant 2026-AD011016047R1) on the Jean Zay supercomputer.