policy-vs-other-v2
A fine-tuned version of xlm-roberta-base that classifies whether a social media post by a political party is about a policy issue or not.
It was developed for PartySOME, a comparative dataset of European parties' social media activity, as the first step of a two-step issue classification. Many party posts have no policy content (campaign events, rallies, greetings, personal content), but issue classifiers such as the Comparative Agendas Project (CAP) models assign a topic to every text. Filtering out non-policy posts first, then classifying the remaining ones with a CAP model, reduces this noise.
Labels
policy issue: the post addresses a policy issueother: it does not
Training data
5,895 manually annotated posts from parties' Facebook, Instagram and Twitter accounts in 18 European countries (Austria, Belgium, Czechia, Denmark, Finland, France, Germany, Hungary, Ireland, Italy, the Netherlands, Norway, Poland, Portugal, Spain, Sweden, Switzerland and the United Kingdom), in their original languages. The annotation criteria are described in the supplementary material of the PartySOME paper (Section B.1).
The posts were split into training (4,422), validation (589) and test (884) sets, stratified by country.
How to use
from transformers import pipeline
classifier = pipeline("text-classification", model="mljn/policy-vs-other-v2", truncation=True)
classifier("The new government policy on taxation was announced today.")
In PartySOME, posts predicted as policy issue are then classified into CAP topics with poltextlab/xlm-roberta-large-pooled-cap-v4.
Evaluation
Results on the held-out test set (884 posts), with policy issue as the positive class:
| Metric | Value |
|---|---|
| Accuracy | 0.893 |
| Precision | 0.905 |
| Recall | 0.914 |
| F1 | 0.910 |
By platform:
| Platform | Posts | Accuracy | F1 |
|---|---|---|---|
| 304 | 0.891 | 0.919 | |
| 282 | 0.908 | 0.927 | |
| 298 | 0.879 | 0.875 |
By country, F1 ranges from 0.77 (Italy) to 0.98 (Denmark), on about 49 test posts per country.
Limitations
- Trained on posts by European political parties. Performance on other kinds of political text, or on other regions, has not been tested.
- Country-level estimates rest on about 49 test posts each and should be read as indicative.
- Whether a post "is about a policy issue" involves judgement calls, for example for posts mixing campaign content with policy statements.
Training hyperparameters
- learning_rate: 2e-05
- train_batch_size: 8
- gradient_accumulation_steps: 2
- eval_batch_size: 16
- weight_decay: 0.01
- optimizer: AdamW
- num_epochs: 4 (best checkpoint selected on validation F1)
- seed: 1234
Citation
If you use this model, please cite:
Jan, Malo, and Luis Sattelmayer. "Issue Competition on Social Media: A New Comparative Dataset on Parties' Issue Agendas." SocArXiv. https://osf.io/preprints/socarxiv/x47qr_v1