This is a fine-tuned Deberta model to detect human values in arguments. The model is part of the ensemble that was the best-performing system in the SemEval2023 task: Detecting Human Values in arguments It was trained and tested on a dataset of 9324 annotated arguments. The whole ensemble system achieved a F1-Score of 0.56 in the competiton. This model achieves a F1-Score of 0.55. Code for retraining the ensemble is accessible in this repo
Model Usage
This model is built on custom code. So the inference api cannot be used directly. To use the model please follow the steps below...
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained("tum-nlp/Deberta_Human_Value_Detector")
trained_model = AutoModelForSequenceClassification.from_pretrained("tum-nlp/Deberta_Human_Value_Detector", trust_remote_code=True)
example_text ='We should ban whaling because whales are a species at the risk of distinction'
encoding = tokenizer.encode_plus(
example_text,
add_special_tokens=True,
max_length=512,
return_token_type_ids=False,
padding="max_length",
return_attention_mask=True,
return_tensors='pt',
)
with torch.no_grad():
test_prediction = trained_model(encoding["input_ids"], encoding["attention_mask"])
test_prediction = test_prediction["output"].flatten().numpy()
Prediction
To make a prediction and map the the outputs to the correct labels. During the competiton a threshold of 0.25 was used to binarize the output.
THRESHOLD = 0.25
LABEL_COLUMNS = ['Self-direction: thought','Self-direction: action','Stimulation','Hedonism','Achievement','Power: dominance','Power: resources','Face','Security: personal',
'Security: societal','Tradition','Conformity: rules','Conformity: interpersonal','Humility','Benevolence: caring','Benevolence: dependability','Universalism: concern','Universalism: nature','Universalism: tolerance','Universalism: objectivity']
print(f"Predictions:")
for label, prediction in zip(LABEL_COLUMNS, test_prediction):
if prediction < THRESHOLD:
continue
print(f"{label}: {prediction}")
Citation
@inproceedings{schroter-etal-2023-adam,
title = "{A}dam-Smith at {S}em{E}val-2023 Task 4: Discovering Human Values in Arguments with Ensembles of Transformer-based Models",
author = "Schroter, Daniel and
Dementieva, Daryna and
Groh, Georg",
editor = {Ojha, Atul Kr. and
Do{\u{g}}ru{\"o}z, A. Seza and
Da San Martino, Giovanni and
Tayyar Madabushi, Harish and
Kumar, Ritesh and
Sartori, Elisa},
booktitle = "Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.semeval-1.74",
doi = "10.18653/v1/2023.semeval-1.74",
pages = "532--541",
abstract = "This paper presents the best-performing approach alias {``}Adam Smith{''} for the SemEval-2023 Task 4: {``}Identification of Human Values behind Arguments{''}. The goal of the task was to create systems that automatically identify the values within textual arguments. We train transformer-based models until they reach their loss minimum or f1-score maximum. Ensembling the models by selecting one global decision threshold that maximizes the f1-score leads to the best-performing system in the competition. Ensembling based on stacking with logistic regressions shows the best performance on an additional dataset provided to evaluate the robustness ({``}Nahj al-Balagha{''}). Apart from outlining the submitted system, we demonstrate that the use of the large ensemble model is not necessary and that the system size can be significantly reduced.",
}