Whisper Bengali medium, GGML q5_1
bengaliAI/tugstugi_bengaliai-asr_whisper-medium converted to GGML and quantised to q5_1 for whisper.cpp, for use on Android.
An optional higher-accuracy alternative to the 190 MB small model, not a replacement for it.
Measured
FLEURS bn_in, 128 clips, both models scored in one run through whisper.cpp
with the settings the app actually uses - no_timestamps, greedy decoding,
common-script romanisation before scoring. The small model measured here is the
q5_1 file the app ships, not a reconversion.
| Model | Size | WER | CER |
|---|---|---|---|
| small q5_1 | 190 MB | 43.4% | 23.9% |
| medium q5_1 | 587 MB | 15.9% | 4.6% |
Both figures had to improve for this to be published. That rule exists because a Hinglish fine-tune once won on one axis while scoring three times worse on the other, and was nearly shipped on the strength of the half it won.
Cost
About 200 seconds per note on a Realme RMX2170 (2 big cores at 2.3 GHz), against 33 for the small model - roughly six times the wait. Whisper pads every input to a 30-second window, so that is per note rather than per second of speech: a four-second note costs the same as a twenty-nine-second one.
Measured, not estimated: five runs on device at the settings above gave 193, 193, 201, 220 and 221 seconds. An earlier figure of 89 seconds appeared in these cards and does not reproduce. It is not memory pressure - available RAM held at 2.7 GB throughout. A medium is about 3.6x the arithmetic of a small model and costs 6x the time, which points at memory bandwidth on a phone.
The same figure applies to every language here: one architecture at one size, all exactly 586,572,019 bytes.
A benchmark that disagrees
Scored on three corpora, the shipping small model and this one:
| Corpus | small q5_1 | medium q5_1 |
|---|---|---|
FLEURS bn_in | 43.4% | 15.9% |
| Kathbath (AI4Bharat IndicSUPERB) | 32.3% | 10.3% |
| imonghose/bengali-asr-data | 9.2% | 10.2% |
The third row is why this model was briefly rejected. Measured only against it, the medium looks a point worse and the gain looks like benchmark contamination. But the anomaly there is the SMALL model's 9.2%, against 32-43% on two unrelated corpora - that dataset appears to overlap its training data, which flattered the baseline rather than the candidate.
Worth stating because the same trap catches anyone testing an Indic model: one control corpus is not a control. It took a third to show which of two disagreeing benchmarks was the strange one.
Settings
no_timestamps is required. These fine-tunes are trained with timestamp
prediction disabled, so their timestamp tokens are untrained and whisper.cpp
derails when it trusts them - 47.5% WER against 14.9% on the small Hindi model,
with the decoder abandoning the audio for fluent unrelated text.
Attribution
Base model Apache-2.0, credit to its authors. Converted with whisper.cpp's
convert-h5-to-ggml.py; token table verified byte for byte against ggerganov's
ggml-small.