wavlm-large-emotion-vad — dimensional speech emotion (Valence / Arousal / Dominance)
WavLM-large fine-tuned end-to-end for dimensional emotion regression on naturalistic speech: audio in → continuous Valence / Arousal / Dominance out. Built as the culmination of a two-phase solo project whose full record (including the handcrafted-feature phase this model made obsolete) is at github.com/Nikhiljangra07/voice-emotion-engine.
Headline result
Evaluated on MSP-Podcast v2.0 Test1: ~46k held-out clips from speakers never seen in training (speaker-independent), metric = Concordance Correlation Coefficient:
| dimension | CCC |
|---|---|
| Valence | 0.705 |
| Arousal | 0.714 |
| Dominance | 0.626 |
| mean | 0.681 |
The point of this model is valence: the handcrafted-acoustic-feature literature treats valence as nearly unrecoverable from audio alone. This project measured that ceiling directly — a handcrafted pipeline built first reached CCC 0.06, then 0.35 after diagnosis — and broke it by fine-tuning the self-supervised representation end-to-end: 0.06 → 0.35 → 0.705. The lever was the representation, not the features.
What's in this repo
| File | Contents |
|---|---|
model.safetensors + config.json | Fine-tuned WavLM-large backbone (WavLMModel) |
head.pt | Regression head (mean-pool → 256-proj → 3 outputs), PyTorch state dict |
train.log | Full training log — 3 epochs, per-epoch dev CCC, final Test1 numbers |
Usage
import torch, torchaudio
from transformers import WavLMModel
backbone = WavLMModel.from_pretrained("Nikhil0097/wavlm-large-emotion-vad")
head = torch.load("head.pt") # download from this repo; see GitHub for the exact head module
# 16 kHz mono in → backbone hidden states → mean-pool → head → [valence, arousal, dominance]
The exact head module, preprocessing, and a ready CLI (audio file in → VAD JSON out) are in
the GitHub repo, which uses this
checkpoint as its inference model.
Training data & license
Trained on MSP-Podcast v2.0 (Lotfian & Busso; Busso et al. 2025, arXiv:2509.09791) under its academic license. The dataset is not included and cannot be redistributed — request access from UT-Dallas. Because the training data is research-licensed, this model is released CC-BY-NC-SA 4.0 (non-commercial).
Honest scope
- Naturalistic speech (podcast-style, English). Acted-corpus performance does not transfer either direction — the project measured this.
- Dominance is the weakest axis (0.626), consistent with the literature.
- Single-model checkpoint, no ensembling; numbers are from one training run whose full log is in this repo.