Multi-label classification of cloud-gaming churn feedback
Fine-tuned distilbert-base-multilingual-cased that tags the technical issue a user complains
about in the free-text comment left when cancelling a cloud-gaming subscription. A single comment
may describe several problems at once, so this is a multi-label task over 6 classes.
Comments are short (median 73 characters) and multilingual — no translation step is used.
👉 Live demo: multilingual-game-feedback-classification.streamlit.app
Labels
| label | meaning |
|---|---|
ping_latency | network delay, high ping, input lag |
frames_drop | low FPS, stuttering, unstable stream |
unable_launch | the game or service does not start |
mouse_keyboard_headset | peripherals: mouse, keyboard, headset, controller |
game_bug | a defect inside the game itself |
failed_save | progress is lost or not saved |
Usage
The model outputs 6 independent sigmoid probabilities. Do not use a 0.5 threshold — per-class
thresholds were tuned on the validation set and are shipped in inference_config.json. They give
macro-F1 0.620 on the test set versus 0.581 with a plain 0.5 threshold.
import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer
REPO = "dar1bi/cloud-gaming-feedback-multilabel"
tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()
config = json.load(open(hf_hub_download(REPO, "inference_config.json")))
text = "Constant micro stutter and very low fps in The Finals"
encoded = tokenizer(text, truncation=True, padding="max_length",
max_length=config["max_len"], return_tensors="pt")
with torch.no_grad():
probabilities = torch.sigmoid(model(**encoded).logits).numpy()[0]
labels = [label for label, probability, threshold
in zip(config["labels"], probabilities, config["thresholds"])
if probability >= threshold]
print(labels) # ['frames_drop']
Training
- Base model:
distilbert-base-multilingual-cased - Objective:
BCEWithLogitsLosswithpos_weightto compensate for class imbalance - Hyperparameters: learning rate 8e-5 (grid over 2e-5 / 5e-5 / 8e-5), 5 epochs with best-epoch selection by validation macro-F1, batch size 16, max sequence length 64
- Split: iterative-stratified multi-label split 70 / 15 / 15 — 2760 / 584 / 612 examples
- Hardware: Apple MPS, ~290 s per configuration
Evaluation
Primary metric is macro-F1 — it weights rare classes equally with frequent ones.
| split | macro-F1 | micro-F1 | macro ROC-AUC | mAP |
|---|---|---|---|---|
| validation | 0.681 | 0.717 | 0.828 | 0.727 |
| test | 0.620 | 0.678 | 0.817 | 0.685 |
Per-class F1 on the test set: ping_latency 0.78, unable_launch 0.70,
mouse_keyboard_headset 0.67, frames_drop 0.62, game_bug 0.49, failed_save 0.46.
The model was selected on the validation set among seven approaches, from TF-IDF with classical classifiers to a zero-shot LLM. A most-frequent-class baseline scores 0.119 macro-F1.
Limitations
- Noisy labels. Labels are self-reported by users in the cancellation form, so they are neither consistent nor mutually exclusive. Part of what looks like model error is actually label noise.
- Rare classes.
game_bugandfailed_saveare recognised in fewer than half of real cases — they are infrequent in the data and semantically overlap with the other classes. - Very short texts. Comments under 30 characters often carry too little signal.
- Single split, single seed. Differences of 0.01–0.02 macro-F1 are within noise.
Training data
Internal user feedback of a cloud-gaming service, collected at subscription cancellation (4 616 records, 3 956 after cleaning). The dataset is confidential and is not published.