Model description
An xlm-roberta-large model fine-tuned on ~1,7 million annotated statements contained in the Manifesto Corpus (version 2024a). The model can be used to categorize any type of text into 56 different political topics according to the Manifesto Project's coding scheme (Handbook 4). It works for all languages the xlm-roberta model is pretrained on (overview), just note that it will perform best for the 38 languages contained in the Manifesto Corpus:
| armenian | bosnian | bulgarian | catalan | croatian |
| czech | danish | dutch | english | estonian |
| finnish | french | galician | georgian | german |
| greek | hebrew | hungarian | icelandic | italian |
| japanese | korean | latvian | lithuanian | macedonian |
| montenegrin | norwegian | polish | portuguese | romanian |
| russian | serbian | slovak | slovenian | spanish |
| swedish | turkish | ukrainian |
How to use
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained("manifesto-project/manifestoberta-xlm-roberta-56policy-topics-sentence-2024-1-1", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
sentence = "We will restore funding to the Global Environment Facility and the Intergovernmental Panel on Climate Change, to support critical climate science research around the world"
inputs = tokenizer(sentence,
return_tensors="pt",
max_length=200, #we limited the input to 200 tokens during finetuning
padding="max_length",
truncation=True
)
logits = model(**inputs).logits
probabilities = torch.softmax(logits, dim=1).tolist()[0]
probabilities = {model.config.id2label[index]: round(probability * 100, 2) for index, probability in enumerate(probabilities)}
probabilities = dict(sorted(probabilities.items(), key=lambda item: item[1], reverse=True))
print(probabilities)
# {'501 - Environmental Protection: Positive': 67.56, '411 - Technology and Infrastructure': 14.03, '107 - Internationalism: Positive': 13.58, '416 - Anti-Growth Economy: Positive': 2.24...
predicted_class = model.config.id2label[logits.argmax().item()]
print(predicted_class)
# 501 - Environmental Protection: Positive
Model Performance
The model was evaluated on a test set of 200,920 annotated manifesto statements.
Overall
| Accuracy | Top2_Acc | Top3_Acc | Precision | Recall | F1_Macro | MCC | Cross-Entropy | |
|---|---|---|---|---|---|---|---|---|
| Sentence Model | 0.57 | 0.73 | 0.81 | 0.48 | 0.43 | 0.45 | 0.55 | 1.47 |
| Context Model | 0.64 | 0.81 | 0.88 | 0.55 | 0.52 | 0.53 | 0.63 | 1.15 |
Citation
Please cite the model as follows:
Burst, Tobias / Lehmann, Pola / Franzmann, Simon / Al-Gaddooa, Denise / Ivanusch, Christoph / Regel, Sven / Riethmüller, Felicia / Weßels, Bernhard / Zehnter, Lisa (2024): manifestoberta. Version 56topics.sentence.2024.1.1. Berlin: Wissenschaftszentrum Berlin für Sozialforschung (WZB) / Göttingen: Institut für Demokratieforschung (IfDem). https://doi.org/10.25522/manifesto.manifestoberta.56topics.sentence.2024.1.1
@misc{Burst:2024,
Address = {Berlin / Göttingen},
Author = {Burst, Tobias AND Lehmann, Pola AND Franzmann, Simon AND Al-Gaddooa, Denise AND Ivanusch, Christoph AND Regel, Sven AND Riethmüller, Felicia AND Weßels, Bernhard AND Zehnter, Lisa},
Publisher = {Wissenschaftszentrum Berlin für Sozialforschung / Göttinger Institut für Demokratieforschung},
Title = {manifestoberta. Version 56topics.sentence.2024.1.1},
doi = {10.25522/manifesto.manifestoberta.56topics.sentence.2024.1.1},
url = {https://doi.org/10.25522/manifesto.manifestoberta.56topics.sentence.2024.1.1},
Year = {2024},