distilbert-multilingual-UITVSMEC
This model is distilbert-base-multilingual-cased fine-tuned for UIT-VSMEC emotion recognition on UIT-VSMEC.
Evaluation protocol
- Dataset size: 6,927 examples.
- Published splits: 5,548 train / 686 development / 693 test.
- Fine-tuning seeds included in the report: [22, 42, 202].
- Training: 3 epoch(s), AdamW, learning rate 2e-05, weight decay 0.01, warmup ratio 0.1.
- Training batch size: 8.
- Maximum sequence length: 256.
- Input mode: raw Vietnamese social-media text.
- The uploaded checkpoint is
best_modelfrom seed 22, selected by development Macro-F1.
Results
Metrics are reported as mean ± sample standard deviation over the completed seeds listed above.
| Metric | Mean ± std |
|---|---|
| Test Macro-F1 | 0.3027 ± 0.0102 |
| Test accuracy | 0.4853 ± 0.0106 |
| Test macro precision | 0.3081 ± 0.0559 |
| Test macro recall | 0.3311 ± 0.0081 |
| Development Macro-F1 | 0.2832 ± 0.0051 |
Per-seed results
| seed | dev_macro_f1 | test_macro_f1 | test_accuracy |
|---|---|---|---|
| 22.000000 | 0.288982 | 0.312447 | 0.489177 |
| 42.000000 | 0.281637 | 0.292191 | 0.473304 |
| 202.000000 | 0.279091 | 0.303494 | 0.493506 |
Label mapping
{
"0": "Anger",
"1": "Disgust",
"2": "Enjoyment",
"3": "Fear",
"4": "Other",
"5": "Sadness",
"6": "Surprise"
}
Usage
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_id = "BaoNhan/distilbert-multilingual-UITVSMEC"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
text = "Đây là nội dung tiếng Việt cần phân loại."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
probabilities = model(**inputs).logits.softmax(dim=-1)[0]
predicted_id = int(probabilities.argmax())
print(model.config.id2label[predicted_id], probabilities.tolist())
Files
aggregate_metrics.json: aggregate metrics and publishing metadata.artifacts/per_seed_results.csv: available completed-seed results.- Other evaluation artifacts are included when present locally.
Limitations
UIT-VSMEC is small and class-imbalanced and reflects Vietnamese social-media language from a particular collection period. Emotion labels are subjective, and predictions must not be treated as psychological assessment.
Dataset citation
@inproceedings{ho-etal-2019-emotion,
title={Emotion Recognition for Vietnamese Social Media Text},
author={Ho, Vong Anh and Nguyen, Duong Huynh-Cong and Nguyen, Danh Hoang and Pham, Linh Thi-Van and Nguyen, Duc-Vu and Nguyen, Kiet Van and Nguyen, Ngan Luu-Thuy},
booktitle={Proceedings of the 16th International Conference of the Pacific Association for Computational Linguistics (PACLING 2019)},
year={2019},
pages={319--333},
url={https://arxiv.org/abs/1911.09339}
}