wavlm-bert-fusion-s-emotion-russian-resd
Speech emotion recognition for Russian over seven classes: anger, disgust, enthusiasm, fear, happiness, neutral, sadness.
Audio tower jonatasgrosman/exp_w2v2t_ru_wavlm_s363 (WavLM/wav2vec2 fine-tuned for Russian ASR) paired with the text encoder DeepPavlov/rubert-base-cased; the model reads the waveform and a transcript together. Trained on Aniemore/resd.
- Parameters: 504M · weights: 1928 MiB (fp32)
- Input: 16 kHz mono waveform and the utterance transcript
- Quantized builds:
wavlm-bert-fusion-s-emotion-russian-resd-quantized— INT8, FP8 and INT4, up to 5.8x smaller on disk at the same score
Results
| Test set | n | UA | WA | macro-F1 |
|---|---|---|---|---|
| RESD test | 280 | 0.7959 | 0.7964 | 0.7941 |
| Dusha podcast test | 12079 | 0.3497 | 0.0700 | 0.0920 |
| CAMEO test | 5187 | 0.2210 | 0.2410 | 0.1983 |
| RESD test, transcript blanked | 280 | 0.7959 | 0.7964 | 0.7941 |
The last row is the audio branch alone. Production rarely has a gold transcript, and the gap between 0.794 with one and 0.794 without is what a transcript is worth here.
Usage
import torch, librosa
from transformers import AutoConfig, AutoTokenizer, AutoModel
repo = "Aniemore/wavlm-bert-fusion-s-emotion-russian-resd"
cfg = AutoConfig.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
tok = AutoTokenizer.from_pretrained(repo)
# Resample to 16 kHz. RESD ships at 44.1 kHz, and 44.1 kHz audio
# labelled as 16 kHz is stretched 2.8x in time — the model answers,
# it just answers about other audio.
wav, _ = librosa.load("clip.wav", sr=16000, mono=True)
wav = torch.tensor((wav - wav.mean()) / (wav.std() + 1e-7))[None]
t = tok("расшифровка реплики", return_tensors="pt")
with torch.no_grad():
logits = model(input_ids=t['input_ids'], input_values=wav,
text_attention_mask=t['attention_mask'],
audio_attention_mask=torch.ones_like(wav).long()).logits
probs = logits.softmax(-1)[0]
print({model.config.id2label[i]: round(p.item(), 3) for i, p in enumerate(probs)})
# torchaudio
import torchaudio
wav, sr = torchaudio.load("clip.wav")
wav = torchaudio.functional.resample(wav, sr, 16000).mean(0).numpy()
# torchcodec, the newer decoder
from torchcodec.decoders import AudioDecoder
wav = AudioDecoder("clip.wav", sample_rate=16000).get_all_samples().data.mean(0).numpy()
# straight from the dataset — `datasets` resamples on the column, so
# the mixed 16/44.1 kHz in RESD is handled for you
from datasets import load_dataset, Audio
ds = load_dataset("Aniemore/resd", split="test")
ds = ds.cast_column("speech", Audio(sampling_rate=16000))
wav = ds[0]["speech"]["array"]
Evaluation protocol
Audio resampled to 16 kHz mono, clips capped at 12 s, normalized per utterance, padding masked. UA is macro-averaged recall, WA is accuracy, F1 is macro-averaged. All three test sets went through the same harness, so the rows are comparable to each other.
The RESD split matches fold 1 of EmoBox bit for bit. The top entry there is WavLM-large at WA 56.47 / UA 55.87 / F1 55.82. These numbers are higher, but the training protocol differs — EmoBox freezes the encoder and trains a probe, this is a full fine-tune — so read it as a different recipe on the same split, not as a like-for-like win.
Limitations
RESD is acted and class-balanced. Real speech is neither. The Dusha podcast row above is the honest signal for spontaneous audio, and it is far below the RESD row. Spontaneous Russian is roughly 93% neutral, and a model tuned on balanced acted data over-predicts the emotional classes on it. Measure on your own material, and calibrate the neutral logit if you deploy.
Seven classes only, Russian only, single-speaker clips.
Citation
@misc{aniemore,
author = {Lubenets, Ilya and Davidchuk, Nikita and Amentes, Aleksandr},
title = {Aniemore: an open library for emotion recognition in Russian speech},
url = {https://github.com/Aniemore/Aniemore},
year = {2023}
}
License
MIT.