Whisper-base for Speech Emotion Recognition in Russian on Dusha dataset
Whisper-base encoder with classification head for speech emotion recognition.
Dusha dataset: https://github.com/salute-developers/golos/tree/master/dusha
Multiclass classification into 5 classes:
- angry 0
- sad 1
- neutral 2
- positive 3
- other 4
Model was fine-tuned on full Dusha-crowd with
- augmentations Time Shift, Time Masking and Colored Noise;
- WeightedRandomSampler.
Usage
import torch
import torchaudio
from transformers import WhisperForAudioClassification, WhisperFeatureExtractor
# load model and feature extractor
model = WhisperForAudioClassification.from_pretrained("waveletdeboshir/whisper-base-ser-dusha")
model.eval()
feature_extractor = WhisperFeatureExtractor.from_pretrained("waveletdeboshir/whisper-base-ser-dusha")
# load audio and resample if necessary
wav, sr = torchaudio.load("audio.wav")
if sr != 16000:
wav = torchaudio.functional.resample(wav, sr, 16000)
# compute predictions
features = feature_extractor(wav[0], sampling_rate=16000, return_tensors="pt").input_features
with torch.no_grad():
preds = model(features)
# get emotion and its probability
probs = torch.nn.functional.softmax(preds.logits, dim=-1)
print(f"Predicted emotion: {model.config.id2label[probs.argmax().item()]} with probability {probs.max().item():.4f}")