whisper-medium-afrispeech
Fine-tune of openai/whisper-medium on
AfriSpeech-200 for African-accented English speech recognition, spanning both
general and clinical/medical domains.
- Base model: openai/whisper-medium (769M)
- Training data: Professor/afrispeech-200-16khz — a clean 16 kHz mono build of AfriSpeech-200 (Intron Health): ~200 h, 120 accents, 13 countries, clinical + general.
- Language: English (African accents)
Results (test split, 6,178 clips)
Fine-tuned vs. zero-shot baseline
Zero-shot = openai/whisper-medium evaluated on the same test set with no fine-tuning.
| Metric | Zero-shot (baseline) | Fine-tuned (this model) | Improvement |
|---|---|---|---|
| Overall WER | 43.22% | 20.13% | −23.1 pts (−53%) |
| Clinical WER | 50.55% | 27.47% | −23.1 pts (−46%) |
| General WER | 36.09% | 12.98% | −23.1 pts (−64%) |
Fine-tuning more than halves WER across the board.
WER by domain
| Domain | WER | n |
|---|---|---|
| General | 12.98% | 2,670 |
| Clinical | 27.47% | 3,508 |
| Overall | 20.13% | 6,178 |
Clinical speech remains ~2× harder than general — medical terminology (drug names, conditions, dosages) drives most of the error, so domain matters when reporting WER.
WER by accent (accents with n ≥ 25)
Performance varies widely across accents (~7% to ~45%):
| Best accents | WER | Hardest accents | WER | |
|---|---|---|---|---|
| okirika | 7.4% | khana | 44.6% | |
| afrikaans | 8.0% | nyandang | 44.1% | |
| brass | 8.6% | |||
| ikwere | 8.9% | |||
| twi | 10.3% |
The hardest accents are mostly smaller, under-represented ones — an important
coverage/equity consideration. Full per-accent numbers: eval_wer_breakdown.csv.
Usage
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
model = WhisperForConditionalGeneration.from_pretrained("Professor/whisper-medium-afrispeech")
processor = WhisperProcessor.from_pretrained("Professor/whisper-medium-afrispeech")
# audio: a 16 kHz mono waveform (numpy array)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
ids = model.generate(inputs.input_features, language="english", task="transcribe")
print(processor.batch_decode(ids, skip_special_tokens=True)[0])
Training
- Objective: full fine-tune (all parameters), 3 epochs
- Precision / hardware: bf16 + gradient checkpointing on a single NVIDIA A40 (48 GB)
- Batch: 32 effective (16 × 2 grad-accum) · LR: 1e-5 (500 warmup, linear decay) · best model by eval WER
- Data handling: on-the-fly log-Mel feature extraction; clips > 30 s filtered out; labels truncated to Whisper's 448-token decoder cap.
Limitations & biases
- Clinical error rate is still elevated (~27%) — verify medical terms; not for unsupervised clinical use.
- Large accent disparity (~7–45% WER) — weaker on under-represented accents.
- ≤ 30 s audio; English only; inherits Whisper's general biases.
Dataset, license & attribution
Trained on AfriSpeech-200 by Intron Health, released under CC-BY-NC-SA-4.0 (non-commercial, share-alike, attribution). This derivative carries the same license.
@article{olatunji2023afrispeech,
title={AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASR},
author={Olatunji, Tobi and others},
journal={Transactions of the Association for Computational Linguistics},
year={2023}
}