whisper-large-v3-turbo-rinpoche
Tibetan speech-to-text for Garchen Rinpoche's speech, fine-tuned from
openai/whisper-large-v3-turbo. Encoder-decoder, 32 encoder / 4 decoder layers,
51,866-token multilingual BPE vocabulary. generation_config.json already sets
language="bo" and task="transcribe".
- Task: automatic speech recognition
- Language: Tibetan (
bo) - Input: 16 kHz mono audio, padded to whisper's 30 s window
- Weights: fp16, single
model.safetensors(1.6 GB) - ⚠️ Recommended segment length: under ~9.5 s — see Limitations
Usage
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
REPO = "GarchenArchive/whisper-large-v3-turbo-rinpoche"
processor = WhisperProcessor.from_pretrained(REPO)
model = WhisperForConditionalGeneration.from_pretrained(
REPO, dtype=torch.float16
).to("cuda").eval()
def transcribe(waveform, sampling_rate=16000):
feats = processor(waveform, sampling_rate=sampling_rate, return_tensors="pt")
with torch.no_grad():
ids = model.generate(feats.input_features.to("cuda", torch.float16))
return processor.tokenizer.decode(
ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False
)
Pass clean_up_tokenization_spaces=False when decoding: the default cleanup is
designed for WordPiece and is destructive for BPE output.
Run inference in fp16. The fp32 master OOMs a 12 GB card during beam search, in the KV-cache reorder.
Evaluation
Measured on the test split of GarchenArchive/stt-rinpoche-202609 (1,394
utterances). That split contains the 554 utterances of the earlier
stt-rinpoche-202605 test split verbatim, so the middle column below is
directly comparable to this model's previous release and to
the ModelFusion benchmark.
CER via jiwer; WER and SER (syllable error rate) via
tibetan_wer, micro averages.
| Decoding | full test (1,394) | 202605 subset (554) | round-2 rows (840) |
|---|---|---|---|
| Greedy | CER 0.2109 | 0.2275 | 0.1889 |
| Beam search (5) | CER 0.2022 | 0.2192 | 0.1796 |
| Pre-segmented to ≤9 s | CER 0.1957 | 0.1993 | 0.1909 |
Beam 5 on the full split: WER 0.3280, SER 0.3284.
Change from the previous release
On the same 554 utterances, against the previous checkpoint:
| previous | this release | |
|---|---|---|
| greedy CER | 0.2327 | 0.2275 |
| beam-5 CER | 0.2257 | 0.2192 |
| pre-segmented CER | 0.2046 | 0.1993 |
A consistent ~2–3 % relative improvement.
Fused with a CTC model
Reconciling this model's output with
GarchenArchive/wav2vec2-xls-r-300m-rinpoche
via an LLM substantially beats either alone — CER 0.1633, WER 0.2804,
SER 0.2846 on the full split (on the 554-utterance 202605 subset: CER 0.1600,
SER 0.2681). The CTC input here is decoded with its KenLM over the raw
vocabulary, with spaces kept — see that model's card. The two models fail
differently, which is what makes the combination work.
Limitations
Output length ceiling. Whisper's decoder is capped at 448 positions and Tibetan tokenises at roughly 43–47 tokens per second of speech in this vocabulary, so audio beyond ~9.5 s cannot fit. The failure is silent: the model emits a well-formed EOS and stops early. You cannot detect it by looking for a missing EOS, because whisper's pad and EOS token ids are identical. Detect it by comparing output length against clip duration.
This dominates the model's error profile, and the effect is visible directly in the deletion counts by subset:
| greedy decoding | mean clip | deletions | insertions |
|---|---|---|---|
| 202605 rows | 7.0 s | 1,983 | 826 |
| round-2 rows | 4.0 s | 795 | 568 |
Same model, same decoding — deletion-dominance is a property of clip length, not of the model. Segment your audio short and it largely disappears.
Beam search returns a degenerate n-best. With num_beams=5, num_return_sequences=5, the returned hypotheses have been observed to be
identical with identical sequences_scores. Beam search itself works — rank 0
differs from greedy — but the n-best list is not usable for reranking.
Domain. Fine-tuned on a single speaker in a specific register. Prior in-house experiments found that further fine-tuning on general Tibetan corpora (Kham dialect, NICTTIB1) degrades performance substantially.
Training
Continued from the previous release rather than re-finetuned from the parent,
on GarchenArchive/stt-rinpoche-202609 (train split: 14,426 utterances,
21.9 h).
| method | LoRA, r=32, α=64, on q/k/v/o + fc1/fc2 |
| trainable | 27.9 M of 836.7 M (3.33 %) |
| learning rate | 1e-4, linear, 500 warmup steps |
| batch | 8, gradient checkpointing, bf16 autocast |
| selection | best eval CER (step 3,000 of 6,000) |
Adapters were merged into the base weights before release, so this repo is a
drop-in replacement requiring no peft at inference.
LoRA rather than a full fine-tune because full AdamW on 809 M parameters needs ~12.9 GB before activations, and the training card has 11.6 GB usable. The 1e-4 LoRA learning rate is not comparable to the 4.375e-6 used for full fine-tuning: adapters initialise at zero and must travel the whole distance.
Training targets longer than the 448-token budget were split at Tibetan clause
boundaries (།) rather than truncated. Truncation is what teaches the
early-EOS behaviour above; 951 over-length rows became 1,914 in-budget pieces.
Transcriber annotations that were never spoken (❓, parenthesised alternate
readings, ༼ ༽) were stripped from the training text.