whisper-small-tibetan
Fine-tuned openai/whisper-small for Lhasa-Tibetan (Central Tibetan) speech recognition.
What this is
A full fine-tune of Whisper-small on a merged corpus of four Lhasa-Tibetan speech sources (read news + natural/spontaneous speech). Zero-shot Whisper is near-useless on Tibetan (>100% WER, heavy hallucination); this fine-tune recovers usable transcription across both read and natural speech domains.
Results
| Metric | Value |
|---|---|
| CER | 11.67% (beam-5, 300-sample held-out speaker test slice) |
| syllable-WER | 16.84% |
Measured on a speaker-disjoint test slice drawn from the merged corpus (speakers the model
never saw). Note this is a 300-sample slice, not the full 25,773-clip test split — the full
number may differ by ~1–2 points. The test set is mixed-domain (read news + natural
speech), which is intrinsically harder than the read-news-only set used in prior stages;
the 11.67% is therefore a stronger result than a direct numeric comparison suggests.
CER (character error rate) is the primary metric: Tibetan is written without word-internal
spaces, so WER is near-meaningless; the reported WER is syllable-based (segmented on the
tsheg ་).
Training progression:
| Model | Test CER |
|---|---|
| Zero-shot whisper-small | >100% WER (hallucination) |
| NICT-Tib1 only (20 spk, 33.5 h) | 22.21% (3-speaker test) |
| Combined (71 spk, 65 h), stage-1 | 14.01% |
| Combined + stage-2 (half LR) | 13.62% |
| 4-corpora merge (359 spk, ~267 h), 3 epochs | 11.67% |
Training data
Four corpora merged and orthographically normalized (one normalize_text() applied in both
prep and metrics):
- NICT-Tib1 — 33.5 h Lhasa-Tibetan read speech, CC BY 4.0, 20 speakers.
- XBMU-Lhasa31 — 31.6 h Lhasa-Tibetan read speech, CC BY-NC-SA 4.0, 51 speakers.
- OpenPecha — ~148k examples of natural/spontaneous speech (teachings, podcasts).
- TIBMD Ü-Tsang — ~41k examples, Central Tibetan (Ü-Tsang) region.
Merged: ~231k examples / ~267 h, speaker-disjoint split — train 185,184 / validation 19,844 / test 25,773 (359 / 77 / 77 speakers).
Intended uses & limitations
- Private / hobby / research use only. The XBMU-Lhasa31 component is CC BY-NC-SA 4.0 (non-commercial), which constrains the resulting model's training data to non-commercial use.
- Lhasa / Central Tibetan focus. Expect degradation on other Tibetan dialects (Amdo, Khams) and on highly colloquial speech outside the corpus domains.
- 30-second chunk limit. Whisper processes ≤30 s per chunk. Longer audio must be split (VAD or fixed chunking) before transcription.
- Not production-grade. At ~11.7% CER this is good for rough transcription / gisting, not authoritative text generation.
Usage
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch
processor = WhisperProcessor.from_pretrained("tenzin/whisper-small-tibetan")
model = WhisperForConditionalGeneration.from_pretrained("tenzin/whisper-small-tibetan")
# audio: 16 kHz mono waveform
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").input_features
forced = processor.tokenizer.get_decoder_prompt_ids(language="bo", task="transcribe")
with torch.no_grad():
ids = model.generate(inputs, forced_decoder_ids=forced, num_beams=5, max_new_tokens=444)
text = processor.tokenizer.batch_decode(ids, skip_special_tokens=True)[0]
Note max_new_tokens=444 (not 448) — Whisper's 4 prompt tokens count against the 448 position
limit.
For faster inference, a CTranslate2 (faster-whisper) conversion is recommended — see
tenzin/whisper-small-tibetan-ct2 (when published) for a ~4× speedup.
Training procedure
Full fine-tune (not LoRA — PEFT is incompatible with Whisper's input_features encoder input).
| Hyperparameter | Value |
|---|---|
| Base model | openai/whisper-small (244M) |
| Precision | bf16 |
| Learning rate | 1e-5 (warmup peak) |
| Effective batch | 16 (16 × grad-accum 1) |
| Epochs | 3 |
| Warmup | 300 steps |
| Label length filter | ≤448 tokens |
| Decode | beam-5, forced Tibetan (language=bo) |
| Gradient checkpointing | off |
Trained on a RunPod RTX PRO 6000 Blackwell (96 GB VRAM, CUDA 13.2, torch 2.8.0+cu132).
License
Training data includes CC BY-NC-SA 4.0 material (XBMU-Lhasa31), so the model's training data is non-commercial. This model is intended for private / non-commercial use.