Whisper-Base-MLA (24 languages) — on-device-tier MLA-Whisper, 62.5% smaller decode KV-cache
openai/whisper-base (74M) with decoder self-attention converted MHA→MLA
(Whisper-MLA, arXiv:2603.00563), recovery-fine-tuned on a
24-language application set — a compact, deployable tier of the 24-language line (whisper-tiny-mla-24lang is smaller still).
from transformers import AutoModelForSpeechSeq2Seq
model = AutoModelForSpeechSeq2Seq.from_pretrained("burakaydinofficial/whisper-base-mla-24lang", trust_remote_code=True) # transformers==4.46.x
Honest sizing note (read first)
Conversion cost GROWS as the model shrinks — measured across the 24-language family: small ≈+0.6 → base ≈+1.1 → tiny ≈+1.9 (approximate per-language medians). At the base tier you pay ≈+1.1 WER (median) for the 62.5% cache cut, and absolute quality is base-tier (much higher WER than small — that is the base model, not MLA). If quality is the priority prefer whisper-small-mla-24lang; this tier is for memory-constrained deployment where the cache cut matters most.
Results (CommonVoice-17 test — Malay from FLEURS, n≤1500/lang (ko 339, no 370, ms 749, vi 1274; rest 1500); cost = paired vs an identically-trained control; CER for th/zh/ja)
| lang | WER% | CER% | conversion cost (WER pts) | sig |
|---|---|---|---|---|
| en | 22.0 | 11.6 | 1.51 | sig |
| de | 30.6 | 11.6 | 0.83 | ns |
| es | 19.6 | 6.8 | 0.77 | sig |
| fr | 33.7 | 14.4 | 0.80 | sig |
| it | 31.9 | 9.4 | 1.39 | sig |
| pt | 29.0 | 10.8 | 0.61 | ns |
| ru | 29.8 | 9.1 | 1.71 | sig |
| nl | 27.9 | 10.4 | 1.06 | sig |
| pl | 36.0 | 11.0 | 1.60 | sig |
| id | 39.9 | 14.1 | 1.31 | sig |
| tr | 40.2 | 12.2 | 1.39 | sig |
| hi | 37.0 | 18.7 | 0.63 | sig |
| ms | 36.7 | 12.8 | 1.65 | sig |
| sv-SE | 44.3 | 16.8 | 1.10 | sig |
| th | 66.9 | 24.0 | 0.17 | ns |
| zh-CN | 43.7 | 25.7 | 0.54 | ns |
| cs | 54.1 | 15.8 | 0.85 | sig |
| vi | 44.5 | 22.5 | 1.31 | sig |
| fi | 53.6 | 12.2 | 2.61 | sig |
| el | 55.3 | 20.7 | 3.69 | sig |
| da | 56.9 | 23.8 | 0.91 | ns |
| ja | n/a | 33.9 | -0.42 | ns |
| nn-NO | 69.4 | 27.0 | 2.07 | sig |
| ko | 60.6 | 29.1 | -1.57 | ns |
Thin-CommonVoice languages (Korean, Norwegian, Danish, Greek, Finnish, Czech, Vietnamese) are the weakest — a DATA limit, not an MLA one. At this base tier several are near-floor (WER: nn-NO 69.4, ko 60.6, da 56.9, el 55.3) — reported for transparency, marginal-to-unusable; at that error level the paired "conversion cost" is dominated by metric noise, not MLA. So restrict the size-cost curve to the usable-WER languages. Per-language cost also depends on the recovery mix — a language's cost here differs from the same language in whisper-base-mla-cv11 (the 11-language model); expected (mix composition moves the low-rank fit), not irreproducibility. Malay was trained on FLEURS (no CV Malay exists; FLEURS is CC-BY-4.0; its eval-only license respected — audio used for training, never redistributed).
Matched control now published — verify the conversion cost yourself: burakaydinofficial/whisper-base-24lang (trained identically, minus the MHA→MLA conversion). Evaluate both with scripts/validate.py.
Limitations
- What the 62.5% is (cache scope): it is the decode self-attention KV-cache — the part that grows with output length and concurrency. The (larger, encoder-length ~1500-frame) cross-attention/encoder memory is NOT compressed, so single-stream total decode-memory savings are modest; the 62.5% cut compounds at output-length × batch concurrency, which is where it pays off.
- Runtime: requires
trust_remote_code+ transformers==4.46.x (no whisper.cpp/CT2/faster-whisper). - Language coverage: covers these 24 languages; erodes Whisper's others.
- Decoding: greedy-decode evals (beam-5 adds ~1-2 WER on both arms; conversion cost unchanged — measured).
- Domain: consumer-mic read-speech.
- Training: 15k steps, warmup+cosine, encoder frozen both arms, dev-selected, bf16 (weights released as fp16).