Whisper Medium fine-tuned on Indian-accented English (AESRC2020)
openai/whisper-medium (769M parameters) fully fine-tuned on the Indian-accent subset of the
AESRC2020 corpus (Accented English Speech Recognition Challenge 2020; Shi et al.,
ICASSP 2021), via the pengyizhou/accented_english mirror. Part of the
indian-asr-bench benchmark's
fine-tuning capacity study across Whisper sizes.
The evaluation setup is unusually clean for a fine-tuning claim: the test split's 481 speakers are fully disjoint from the 38 training speakers (zero overlap), so the measured improvement is generalization to unseen speakers, not speaker memorization.
Results
Corpus WER on the 1,731-clip Indian-accent test split. The baseline is the
pretrained openai/whisper-medium decoded through the identical HuggingFace chunked pipeline,
so the delta isolates fine-tuning from any decoding or engine difference.
| Metric | Pretrained baseline | This model | Delta |
|---|---|---|---|
| WER, benchmark gold normalizer | 5.63% | 4.48% | -1.15 pp |
| WER, OpenAI Whisper normalizer | 5.26% | 4.18% |
Paired speaker-clustered bootstrap (2,000 resamples over 481 speakers): delta -1.15 pp, 95% CI [-1.55, -0.77], Holm-corrected p = 0.003.
The gain is statistically significant after Holm correction across the three-size family, about a 20% relative WER reduction.
Seed replication
This is a single training run. To check that its gain is not a lucky draw, the identical recipe was retrained from 6 seeds (42-47) and scored on the same speaker-disjoint test set:
| Δ WER vs. pretrained | |
|---|---|
| This checkpoint (1 seed) | -1.15 pp |
| 6-seed mean | -1.22 pp (SD 0.12) |
| 6-seed range | -1.32 to -1.00 pp |
All six improve on the pretrained baseline and the range never approaches zero. This checkpoint's run sits close to the 6-seed mean, so it is representative.
The six checkpoints are published at
theshivam7/whisper-medium-aesrc-indian-english-seeds,
and the per-run numbers are in
finetune_seeds_transcript_clean_per_seed.csv.
Training
- Data:
pengyizhou/accented_english, rows withaccent == "INDIAN", train split: 12,820 clips, 17.5 hours, 38 speakers (dataset revision pinned to4a80d8388f06368a0fa2a325770bec3492cabd3d). - Recipe: full fine-tune, 2,000 steps, effective batch 32 (batch 8 x grad-accum 4), lr 1e-5 with 100 warmup steps, fp16, seed 42, on one NVIDIA A100-40GB.
- Checkpoint selection: best validation WER (OpenAI's
EnglishTextNormalizer), evaluated every 200 steps. - Training script:
finetune/finetune_tiny_small.py.
Limitations
- The validation split used for checkpoint selection shares all 38 speakers with the training split, so selection measures fit, not speaker generalization. The speaker-disjoint test split was never touched during training.
- Single training run; no seed replication, so run-to-run variance is unbounded.
- The upstream mirror states no license for the audio and transcripts. AESRC2020 is Datatang's corpus; check the data-use terms before commercial or publication use of anything derived from the training data.
- Trained and evaluated on short (mean 4.5 s) prompted read speech. Behavior on long-form or spontaneous audio is untested here.
Usage
from transformers import pipeline
asr = pipeline("automatic-speech-recognition",
model="theshivam7/whisper-medium-aesrc-indian-english",
chunk_length_s=30)
print(asr("audio.wav")["text"])
Benchmark context
The full study evaluates nine pretrained ASR systems and six fine-tuned Whisper checkpoints on three Indian-English corpora, with speaker-clustered statistics and a reference-artifact audit. Code, raw transcripts, and every number behind this card: github.com/theshivam7/indian-asr-bench.