sarcasm-distilbert
A fine-tuned distilbert-base-uncased model for classifying news headlines as Sarcastic or Not Sarcastic.
- GitHub repo (training notebook, EDA, baseline comparison): https://github.com/krskumarsatyam777-glitch/news-headline-sarcasm-detector
- Live demo: https://huggingface.co/spaces/satyame639291/news_headline_scarcasm_detector
Model Description
Fine-tuned version of distilbert-base-uncased for binary sequence classification, trained on the News Headlines Dataset for Sarcasm Detection (Misra, 2019) — headlines from The Onion (satirical) paired with headlines from HuffPost (genuine).
| Label | Meaning |
|---|---|
| 0 | Not Sarcastic (genuine, from HuffPost) |
| 1 | Sarcastic (satirical, from The Onion) |
How to Get Started
from transformers import pipeline
classifier = pipeline("text-classification", model="satyame639291/sarcasm-distilbert")
classifier("Shocking, Usain bolt realisies he can use legs for walking")
Training Details
- Base model:
distilbert-base-uncased - Epochs: 3 | Learning rate: 2e-5 | Batch size: 32 | Weight decay: 0.01
- Model selection: Best checkpoint by validation F1 (
load_best_model_at_end=True) — epoch 2 was selected as the final model. - Hardware: Single T4 GPU (Google Colab).
Results
Evaluated on a held-out test set (4,276 headlines) not used in training or model selection:
| Model | Accuracy | F1 (weighted) |
|---|---|---|
| TF-IDF + Logistic Regression (default) | 77.34% | 0.7729 |
| TF-IDF + Logistic Regression (tuned) | 77.39% | 0.7736 |
| DistilBERT (this model) | 92.00% | 0.9200 |
Full methodology, EDA, and error analysis: training notebook.
Error patterns: false positives tended to be plain, non-humorous headlines rather than obviously quirky ones; false negatives mostly required outside knowledge (e.g. a public figure's reputation) that isn't present in the text itself.
Limitations
- Labels come from source publication (Onion vs. HuffPost), not human sarcasm annotation — the model may partly learn outlet writing style rather than sarcasm itself.
- Not validated on social media, reviews, conversational text, or non-English text.
- Sarcasm requiring external/cultural knowledge is harder to detect than sarcasm signaled by wording alone.
Contact
Satyam — GitHub | Hugging Face