Indic Whisper with a character CTC head: transcript and word timestamps in one pass
Whisper large-v2 fine-tuned for Hindi, with a character-level CTC head over the same encoder. Audio goes through the encoder once: the attention decoder reads the result to produce the transcript, and the CTC head reads the same tensor to place word boundaries by forced alignment at 20 ms resolution. No second model and no second encoder pass, unlike pipelines that run a separate wav2vec2 aligner.
This is configuration M4 of the accompanying paper:
📄 Timestamped Hindi Speech Transcription using Whisper Sanat Kumar Agrawal, Srikanth Raj Chetupalli. Submitted to ICASSP 2027.
| value | |
|---|---|
| WER (IndicVoices-R Hindi test) | 14.6% |
| word-boundary error vs. external aligner | 20.8 ms |
| boundaries within 50 ms | 93.0% |
| CTC head size | 367 KB (one linear layer) |
Usage
python word_timestamps.py clip.wav --model Sanat-agrwl/indic-whisper-ctc-timestamps --head ctc_head_E.pt
Outputs the transcript followed by one line per word with start and end times in seconds.
Calibration
Boundaries from a CTC head are systematically tight: words start about 38 ms late and end
about 100 ms early relative to human annotation. Subtracting those two constants roughly
halves the error, from 74.5 to 43.0 ms mean, measured out-of-sample against human boundaries.
Pass --calibrate to apply them. Because the offsets apply independently, calibrated spans of
adjacent words can overlap slightly.
What the timestamp numbers mean
The 20.8 ms above is agreement with an external wav2vec2 aligner, not accuracy. Measured against human annotation, this head and that aligner are statistically indistinguishable (74.5 vs 72.9 ms) yet agree with each other far more closely than either agrees with a person, because they share a systematic bias. Human inter-annotator agreement on the same task is 27.0 ms. See the annotations release and the paper.
Provenance and licence
Fine-tuned from vasista22/whisper-hindi-large-v2 (Apache-2.0), itself derived from OpenAI Whisper large-v2 (MIT), on IndicVoices-R (CC BY 4.0). Released under Apache-2.0; please attribute all three.
Citation
@article{agrawal2026timestamped,
title = {Timestamped Hindi speech transcription using Whisper},
author = {Agrawal, Sanat Kumar and Chetupalli, Srikanth Raj},
journal = {arXiv preprint arXiv:2610.08847},
year = {2026}
}