Irony detection in English
bertweet-irony
Repository: https://github.com/pysentimiento/pysentimiento/
Model trained with SemEval 2018 dataset Task 3 (Van Hee et all, 2018) for irony detection. Base model is [BERTweet], a RoBERTa model trained in English tweets.
The positive class marks irony, the negative class marks not ironic content.
Results
Results for the four tasks evaluated in pysentimiento. Results are expressed as Macro F1 scores
| Model | sentiment | emotion | hate_speech | irony |
|---|---|---|---|---|
| bert | 69.6 +- 0.4 | 42.7 +- 0.6 | 56.0 +- 0.8 | 68.1 +- 2.2 |
| electra | 70.9 +- 0.4 | 37.2 +- 2.9 | 55.6 +- 0.6 | 71.3 +- 1.8 |
| roberta | 70.4 +- 0.3 | 45.0 +- 0.9 | 55.1 +- 0.4 | 70.4 +- 2.9 |
| robertuito | 69.6 +- 0.5 | 43.0 +- 3.3 | 57.5 +- 0.2 | 73.9 +- 1.4 |
| bertweet | 72.0 +- 0.4 | 43.1 +- 1.8 | 57.7 +- 0.7 | 80.8 +- 0.7 |
Note that for Hate Speech, these are the results for Semeval 2019, Task 5 Subtask B (HS+TR+AG detection)
Citation
If you use this model in your research, please cite pysentimiento, dataset and pre-trained model papers:
@misc{perez2021pysentimiento,
title={pysentimiento: A Python Toolkit for Sentiment Analysis and SocialNLP tasks},
author={Juan Manuel Pérez and Juan Carlos Giudici and Franco Luque},
year={2021},
eprint={2106.09462},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@inproceedings{van2018semeval,
title={Semeval-2018 task 3: Irony detection in english tweets},
author={Van Hee, Cynthia and Lefever, Els and Hoste, V{\'e}ronique},
booktitle={Proceedings of The 12th International Workshop on Semantic Evaluation},
pages={39--50},
year={2018}
}
@inproceedings{nguyen2020bertweet,
title={BERTweet: A pre-trained language model for English Tweets},
author={Nguyen, Dat Quoc and Vu, Thanh and Nguyen, Anh Tuan},
booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
pages={9--14},
year={2020}
}