typhoon-whisper-medium
Thai Whisper-medium with contextual biasing support via a decoder LoRA. Merged base weights + adapter in one repo.
Full fine-tune of openai/whisper-medium on ~10,846 hours of Thai speech (see the Typhoon ASR Real-time technical report).
Results
Character Error Rate (%, lower is better), spaces macro-CER protocol.
| benchmark | CER |
|---|---|
| TVSpeech (570 utts, out-of-domain robustness) | 7.50 |
| gigaspeech2-typhoon test (gs2_bench, 1,000 utts, in-domain) | 4.78 |
Contextual biasing (thai-contextasr-bench, 2,062 utts)
| condition | CER % | entity recall | Δrecall vs none |
|---|---|---|---|
| none | 7.05 | 0.383 | – |
| bias@0 | 4.92 | 0.658 | +0.275 |
| bias@10 | 8.09 | 0.597 | +0.214 |
| dis@10 | 8.13 | 0.375 | -0.008 |
bias@0 = gold entities only (an oracle upper bound). bias@10 = gold plus 10 distractors, the realistic condition. dis@10 = distractors only, measuring over-biasing.
Usage
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
model_id = "typhoon-ai/typhoon-whisper-medium"
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(
model_id, dtype=torch.bfloat16).to("cuda").eval()
feats = processor(audio, sampling_rate=16000, return_tensors="pt").input_features
feats = feats.to("cuda", torch.bfloat16)
ids = model.generate(feats, language="th", task="transcribe", max_new_tokens=440)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])
Contextual biasing
Load the bundled LoRA adapter (contextual biasing lives in the adapter; the
base model alone is a plain Thai ASR), then pass a bias list through Whisper's
initial prompt (<|startofprev|>) in the training carrier format.
⚠️ Use this exact carrier format. The model was trained only with the prompt
"คำศัพท์ที่เกี่ยวข้อง: " + " ".join(entries)(space-joined entries after the carrier phrase). Any other phrasing, separator, or format was never seen in training and will degrade or disable the biasing effect.
from peft import PeftModel
model = PeftModel.from_pretrained(model, model_id, subfolder="adapter")
prompt = "คำศัพท์ที่เกี่ยวข้อง: " + " ".join(entries)
ids = processor.get_prompt_ids(prompt, return_tensors="pt").to(model.device)
out = model.generate(features, language="th", task="transcribe", prompt_ids=ids)
Architectural limit: Whisper's prompt is capped at 224 tokens inside a 448-token decoder window. Lists beyond ~15–20 entries truncate and degrade sharply — this is a property of the architecture, not of this fine-tune. Keep lists short; retrieve the top-k entries per utterance if you have a large vocabulary.
Limitations
- Domain: >95 % of the training corpus is GigaSpeech2-th (Thai YouTube). In-domain accuracy (gs2_bench) is far stronger than out-of-domain robustness (TVSpeech).
- Benchmark overlap: 3.0 % of gs2_bench utterances share boilerplate transcripts (sponsor taglines, programme intros) with training data. No audio clip is reused — id overlap is 0 — and this affects every system trained on GigaSpeech2-th equally, but read the in-domain number with it in mind. TVSpeech is 100 % clean.
- Thai only. Fine-tuned exclusively on Thai; other languages are not preserved.
- 30 s window. Standard Whisper limit; longer audio needs chunking.
Acknowledgements
We thank Kunat Pipatanakul and Potsawee Manakul (wayu-ai) for their valuable feedback.