Wav2Vec2-XLSR-53 Fine-Tuned for Stuttering Detection

A Wav2Vec2ForSequenceClassification model fine-tuned on the SEP-28k-Extended dataset for multi-class stuttering type classification.

This model is part of the SpeechTherapyAgent (VST) system described in the paper:

SpeechTherapyAgent: A Clinician-in-the-Loop AI Virtual Speech Therapist for Personalized and Supervised Therapy Shakeel A. Sheikh, Patrick Marmaroli, Md Sahidullah, Slim Ouni, Fabrice Hirsch, Gonçalo Leal, Björn W. Schuller (under review)

Live demo: vocametrix.com/ai/stuttering-therapy-planning-agent

Model Details

Labels

Performance

Weighted average F1 scores on SEP-28k-Extended (speaker-disjoint split):

Usage

from transformers import AutoModelForAudioClassification, AutoFeatureExtractor
import torch
import torch.nn.functional as F
import librosa

# Load model
model_name = "vocametrix/wav2vec2-xlsr-53-stuttering-classification"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_name)
model = AutoModelForAudioClassification.from_pretrained(model_name)
model.eval()

# Load audio (16 kHz, mono)
audio, sr = librosa.load("your_audio.wav", sr=16000, mono=True)

# Classify
inputs = feature_extractor(audio, sampling_rate=16000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(**inputs).logits
    probs = F.softmax(logits, dim=-1)
    pred_id = torch.argmax(probs, dim=-1).item()
    confidence = probs[0][pred_id].item()
    label = model.config.id2label[pred_id]

print(f"Prediction: {label} ({confidence:.1%})")

Chunked Inference

For longer audio recordings, we recommend splitting into 4-second chunks with 50% overlap (as used in the VST system):

import numpy as np

chunk_duration = 4.0  # seconds
overlap = 0.50
chunk_samples = int(chunk_duration * 16000)
step_samples = int(chunk_samples * (1 - overlap))

results = []
pos = 0
while pos < len(audio):
    end = min(pos + chunk_samples, len(audio))
    chunk = audio[pos:end]
    if len(chunk) < chunk_samples:
        chunk = np.pad(chunk, (0, chunk_samples - len(chunk)))

    inputs = feature_extractor(chunk, sampling_rate=16000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(**inputs).logits
        probs = F.softmax(logits, dim=-1)
        pred_id = torch.argmax(probs, dim=-1).item()
        results.append({
            "start": pos / 16000,
            "end": end / 16000,
            "label": model.config.id2label[pred_id],
            "confidence": probs[0][pred_id].item(),
        })
    pos += step_samples

Inference Speed

Citation

If you use this model in your research, please cite:

@article{sheikh2026speechtherapyagent,
  title={SpeechTherapyAgent: A Clinician-in-the-Loop AI Virtual Speech Therapist for Personalized and Supervised Therapy},
  author={Sheikh, Shakeel A. and Marmaroli, Patrick and Sahidullah, Md and Ouni, Slim and Hirsch, Fabrice and Leal, Gon\c{c}alo and Schuller, Bj\"orn W.},
  journal={Under review},
  year={2026}
}

License

Apache 2.0

Acknowledgements

• Base model: facebook/wav2vec2-large-xlsr-53
• Dataset: SEP-28k / SEP-28k-Extended
• Developed by Vocametrix