rubert-large-emotion-russian-cedr-m7
Emotion detection in Russian text over seven classes, fine-tuned from sberbank-ai/ruBert-large on Aniemore/cedr-m7 (427M parameters).
The head is multi-label: seven independent sigmoids, not a softmax over seven classes, so a sentence may carry more than one emotion or none at all. That is why the headline number below is ROC AUC, which needs no threshold.
Results on CEDR-M7 test
| Metric | Value |
|---|---|
| ROC AUC, macro | 0.9258 |
| ROC AUC, micro | 0.9714 |
| ROC AUC, weighted | 0.9618 |
| Weighted accuracy (top-1) | 0.8395 |
| Unweighted accuracy (macro recall) | 0.6610 |
| macro-F1 @ 0.5 | 0.6724 |
| micro-F1 @ 0.5 | 0.8372 |
| Subset accuracy (exact set) | 0.8135 |
How to read these
ROC AUC is threshold-free: it asks whether the model ranks the sentences carrying an emotion above the ones that do not. macro-F1 is taken at a fixed threshold of 0.5; a different operating point moves it, so tune the threshold on your own data before reading it as a ceiling. Weighted accuracy is top-1: the highest-scoring label counted against the gold set, which is a fair reading here because 1865 of the 1882 test rows carry exactly one label.
disgust has 3 examples in the test split. Its per-class numbers are noise, and they drag both macro averages. The macro figures are reported as measured rather than quietly dropping the class, but weigh them with that in mind.
Evaluated on the official test split, 1882 rows, sources: twitter, LiveJournal and Lenta. Scores come from the model's own id2label order — it differs across this family, and reading the labels in the wrong order silently mislabels everything.
First - you should prepare few functions to talk to model
import torch
from transformers import BertForSequenceClassification, AutoTokenizer
LABELS = ['anger', 'disgust', 'enthusiasm', 'fear', 'happiness', 'neutral', 'sadness']
tokenizer = AutoTokenizer.from_pretrained('Aniemore/rubert-large-emotion-russian-cedr-m7')
model = BertForSequenceClassification.from_pretrained('Aniemore/rubert-large-emotion-russian-cedr-m7')
@torch.no_grad()
def predict_emotion(text: str) -> str:
"""
We take the input text, tokenize it, pass it through the model, and then return the predicted label
:param text: The text to be classified
:type text: str
:return: The predicted emotion
"""
inputs = tokenizer(text, max_length=512, padding=True, truncation=True, return_tensors='pt')
outputs = model(**inputs)
predicted = torch.nn.functional.softmax(outputs.logits, dim=1)
predicted = torch.argmax(predicted, dim=1).numpy()
return LABELS[predicted[0]]
@torch.no_grad()
def predict_emotions(text: str) -> list:
"""
It takes a string of text, tokenizes it, feeds it to the model, and returns a dictionary of emotions and their
probabilities
:param text: The text you want to classify
:type text: str
:return: A dictionary of emotions and their probabilities.
"""
inputs = tokenizer(text, max_length=512, padding=True, truncation=True, return_tensors='pt')
outputs = model(**inputs)
predicted = torch.nn.functional.softmax(outputs.logits, dim=1)
emotions_list = {}
for i in range(len(predicted.numpy()[0].tolist())):
emotions_list[LABELS[i]] = predicted.numpy()[0].tolist()[i]
return emotions_list
And then - just gently ask a model to predict your emotion
simple_prediction = predict_emotion("Какой же сегодня прекрасный день, братья")
not_simple_prediction = predict_emotions("Какой же сегодня прекрасный день, братья")
print(simple_prediction)
print(not_simple_prediction)
Or, just simply use our package (GitHub), that can do whatever you want (or maybe not)
🤗
Citations
@misc{Aniemore,
author = {Артем Аментес, Илья Лубенец, Никита Давидчук},
title = {Открытая библиотека искусственного интеллекта для анализа и выявления эмоциональных оттенков речи человека},
year = {2022},
publisher = {Hugging Face},
journal = {Hugging Face Hub},
howpublished = {\url{https://huggingface.com/aniemore/Aniemore}},
email = {hello@socialcode.ru}
}