Model Card for mDeBERTa Stance Detection

A multilingual stance detection model fine-tuned for detecting political stance towards specific groups in text.

Model Details

Model Description

This model is a fine-tuned mDeBERTa-v3-base that performs stance classification using Natural Language Inference (NLI) to determine whether political text expresses positive, negative, or neutral stance towards specific target groups.

• Developed by: Will Horne, Alona O. Dolinsky and Lena Maria Huber
• Model type: Sequence Classification (NLI-based stance detection)
• Language(s) (NLP): English, German (multilingual)
• Finetuned from model: MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7

Model Sources

• Repository: rwillh11/mdeberta_NLI_stance_NoContext
• Base Model: MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7

Uses

Direct Use

The model is designed for researchers analyzing political discourse and stance towards specific groups in political text, trained and validated using party manifestos. It takes a natural sentence and a target group as input and classifies the stance as positive, negative, or neutral.

Downstream Use

This model can be integrated into larger political text analysis pipelines for:

• Political manifestos analysis
• Group appeals detection in political communication
• Comparative political research across countries and languages

Out-of-Scope Use

This model should not be used for:

• General sentiment analysis (not group-specific)
• Real-time social media monitoring without human oversight
• Making decisions about individuals or groups
• Content moderation without additional validation

Bias, Risks, and Limitations

Technical Limitations

• Trained specifically on political manifesto text; performance may vary on other text types
• Limited to three stance categories (positive, negative, neutral)

Bias Considerations

• Training data consists of political manifestos from specific countries and time periods
• May reflect biases present in political discourse of training data
• Stance classification may vary across different political contexts

Recommendations

Users should be aware that this model:

• Is designed for research purposes in political science
• Should be validated on specific domains before deployment
• May require human oversight for sensitive applications
• Performance may vary across different types of groups and political contexts

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_name = "rwillh11/mdeberta_NLI_stance_NoContext"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Example usage
text = "We will increase funding for schools to better support students."
target_group = "students"

# Create hypotheses for each stance
hypotheses = {
    "positive": f"The text is positive towards {target_group}.",
    "negative": f"The text is negative towards {target_group}.",
    "neutral": f"The text is neutral, or contains no stance, towards {target_group}."
}

# Get predictions for each hypothesis
results = {}
for stance, hypothesis in hypotheses.items():
    inputs = tokenizer(text, hypothesis, return_tensors="pt", truncation=True)
    with torch.no_grad():
        outputs = model(**inputs)
        probs = torch.softmax(outputs.logits, dim=-1)
        entailment_prob = probs[0][0].item()  # Probability of entailment
        results[stance] = entailment_prob

# Select stance with highest entailment probability
predicted_stance = max(results, key=results.get)
print(f"Predicted stance towards '{target_group}': {predicted_stance}")

Training Details

Training Data

The model was trained on political manifesto data containing:

• Languages: English and German
• Text Type: Political manifesto sentences at natural sentence level
• Labels: Three-class stance classification (positive, negative, neutral)
• Groups: Various political target groups (citizens, specific demographics, etc.)
• Original dataset: 7,567 text-group pairs
  • English: 4,110 examples (54.3%)
  • German: 3,457 examples (45.7%)
• Training Size: 12,104 expanded training examples (~6,054 original texts × 2 hypotheses each)
• Test Size: 4,539 expanded test examples (~1,513 original texts × 3 hypotheses each)

Training Procedure

Preprocessing

• Texts tokenized using mDeBERTa tokenizer with max length 512
• NLI format: premise (political text) + hypothesis (stance towards group)
• Each text paired with both true and false hypotheses for binary classification

Training Hyperparameters

• Training regime: Mixed precision training
• Optimizer: AdamW with weight decay
• Learning rate: Optimized via Optuna (range: 1e-5 to 4e-5)
• Weight decay: Optimized via Optuna (range: 0.01 to 0.3)
• Warmup ratio: Optimized via Optuna (range: 0.0 to 0.1)
• Epochs: 10 per trial
• Batch size: 16 (train and eval)
• Trials: 20 total (10 + 10 batches)
• Metric for selection: F1 Macro
• Seed: 42 (deterministic training)

Training Infrastructure

• Hardware: CUDA-enabled GPU
• Framework: Transformers, PyTorch
• Hyperparameter optimization: Optuna
• Deterministic training: All random seeds fixed

Evaluation

Testing Data, Factors & Metrics

Testing Data

• 20% holdout from original dataset
• Multilingual political manifesto sentences

Factors

The model was evaluated across:

• Languages: English and German text
• Additional validation on held out sets: English, German, Dutch, Danish, Spanish, French, Italian, Swedish
• Stance classes: Positive, negative, neutral
• Group types: Various socia-demographic groups

Metrics

Primary metrics used for evaluation:

• F1 Macro: Primary optimization metric (treats all classes equally)
• Balanced Accuracy: Accounts for class imbalance
• Precision/Recall (Macro): Detailed performance measures

Results

Best Model Performance (Trial 19, Epoch 9):

• Accuracy: 0.86
• Balanced Accuracy: 0.81
• Precision: 0.83
• Recall: 0.81
• F1 Macro: 0.81

Additional validation on held-out sets return the following metrics:

English

• Accuracy: 0.86
• Precision: 0.84
• Recall: 0.81
• F1 Macro: 0.82

German (using texts translated from English)

• Accuracy: 0.86
• Precision: 0.85
• Recall: 0.79
• F1 Macro: 0.82

Dutch (using texts translated from English)

• Accuracy: 0.82
• Precision: 0.80
• Recall: 0.74
• F1 Macro: 0.76

Danish (using texts translated from English)

• Accuracy: 0.83
• Precision: 0.80
• Recall: 0.75
• F1 Macro: 0.77

Spanish (using texts translated from English)

• Accuracy: 0.85
• Precision: 0.84
• Recall: 0.77
• F1 Macro: 0.80

French (using texts translated from English)

• Accuracy: 0.84
• Precision: 0.84
• Recall: 0.76
• F1 Macro: 0.79

Italian (using texts translated from English)

• Accuracy: 0.86
• Precision: 0.86
• Recall: 0.77
• F1 Macro: 0.81

Swedish (using texts translated from English)

• Accuracy: 0.82
• Precision: 0.79
• Recall: 0.77
• F1 Macro: 0.78

The model demonstrates strong performance across stance categories with deterministic results confirmed through multiple prediction runs.

Model Examination

The model uses Natural Language Inference to transform stance detection into a binary entailment task:

• For each text-group pair, generates three hypotheses (positive/negative/neutral stance)
• Selects the hypothesis with highest entailment probability
• This approach leverages pre-trained NLI capabilities for stance classification

Environmental Impact

Training involved hyperparameter optimization with 20 trials, each training for 10 epochs.

• Hardware Type: CUDA-enabled GPU
• Hours used: Estimated 10-15 hours (including hyperparameter search)
• Cloud Provider: Google Colab
• Compute Region: Variable
• Carbon Emitted: Not precisely measured

Technical Specifications

Model Architecture and Objective

• Base Architecture: mDeBERTa-v3-base (278M parameters)
• Task: Natural Language Inference for stance detection
• Input: Text pair (political sentence + stance hypothesis)
• Output: Binary classification (entailment/non-entailment)
• Objective: Cross-entropy loss with F1 Macro optimization

Compute Infrastructure

Hardware

• GPU-accelerated training (CUDA)
• Mixed precision training support

Software

• Transformers library
• PyTorch framework
• Optuna for hyperparameter optimization
• scikit-learn for metrics

Citation

If you use this model in your research, please cite:

BibTeX:

@misc{mdeberta_stance_nocontext,
  title={mDeBERTa Stance Detection Model for Political Group Appeals},
  author={Research Team},
  year={2024},
  url={https://huggingface.co/rwillh11/mdeberta_NLI_stance_NoContext}
}

Model Card Authors

Research team studying group appeals in political discourse.

Model Card Contact

For questions about this model, please open an issue in the repository or contact the research team through appropriate academic channels.