whisper-small-afrispeech
Fine-tune of openai/whisper-small on
AfriSpeech-200 for African-accented English speech recognition, spanning both
general and clinical/medical domains.
- Base model: openai/whisper-small (244M)
- Training data: Professor/afrispeech-200-16khz — a clean 16 kHz mono build of AfriSpeech-200 (Intron Health): ~200 h, 120 accents, 13 countries, clinical + general.
- Language: English (African accents)
Results (test split, 6,178 clips)
Fine-tuned vs. zero-shot baseline
Zero-shot = openai/whisper-small evaluated on the same test set with no fine-tuning.
| Metric | Zero-shot (baseline) | Fine-tuned (this model) | Improvement |
|---|---|---|---|
| Overall WER | 49.15% | 22.94% | −26.2 pts (−53%) |
| Clinical WER | 56.28% | 29.84% | −26.4 pts (−47%) |
| General WER | 42.21% | 16.23% | −26.0 pts (−62%) |
Fine-tuning roughly halves WER across the board — a 26-point absolute reduction overall — and cuts clinical error from 56% to 30%.
WER by domain
| Domain | WER | n |
|---|---|---|
| General | 16.23% | 2,670 |
| Clinical | 29.84% | 3,508 |
| Overall | 22.94% | 6,178 |
Clinical speech is ~1.8× harder than general — medical terminology (drug names, conditions, dosages) drives most of the error, so domain matters when reporting WER.
WER by accent (70 accents with n ≥ 25)
Performance varies widely across accents (~8% to ~62%):
| Best accents | WER | Hardest accents | WER | |
|---|---|---|---|---|
| okirika | 8.2% | agatu | 61.6% | |
| brass | 9.4% | mwaghavul | 58.1% | |
| afrikaans | 11.7% | efik | 57.8% | |
| twi | 11.9% | bini | 51.0% | |
| ikwere | 12.1% | idah | 47.7% |
The hardest accents are mostly smaller, under-represented Nigerian accents — an important
coverage/equity consideration. Full per-accent numbers: eval_wer_breakdown.csv.
Usage
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
model = WhisperForConditionalGeneration.from_pretrained("Professor/whisper-small-afrispeech")
processor = WhisperProcessor.from_pretrained("Professor/whisper-small-afrispeech")
# audio: a 16 kHz mono waveform (numpy array)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
ids = model.generate(inputs.input_features, language="english", task="transcribe")
print(processor.batch_decode(ids, skip_special_tokens=True)[0])
Training
- Objective: full fine-tune (all parameters), 3 epochs
- Precision / hardware: bf16 on a single NVIDIA A40 (48 GB)
- Batch: 32 · LR: 1e-5 (500 warmup steps, linear decay) · best model by eval WER
- Data handling: on-the-fly log-Mel feature extraction; clips > 30 s filtered out (Whisper's fixed 30 s window); labels truncated to Whisper's 448-token decoder cap.
- Runtime: ~83 min.
Limitations & biases
- Clinical error rate is high (~30%) — verify medical terms; not for unsupervised clinical use.
- Large accent disparity (~8–62% WER) — much weaker on under-represented accents.
- ≤ 30 s audio — longer inputs are truncated to 30 s by Whisper's feature extractor.
- English only; inherits Whisper's general biases.
Dataset, license & attribution
Trained on AfriSpeech-200, created by Intron Health and released under CC-BY-NC-SA-4.0 (non-commercial, share-alike, attribution). This derivative model carries the same license. Please cite AfriSpeech-200:
@article{olatunji2023afrispeech,
title={AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASR},
author={Olatunji, Tobi and others},
journal={Transactions of the Association for Computational Linguistics},
year={2023}
}