wav2vec2-xls-r-300m-rinpoche

Tibetan speech-to-text for Garchen Rinpoche's speech, fine-tuned from facebook/wav2vec2-xls-r-300m with a CTC head over a 129-token character vocabulary. A 5-gram KenLM (kenlm_5gram.bin) is included for beam-search decoding.

• Task: automatic speech recognition
• Language: Tibetan (bo)
• Input: 16 kHz mono audio
• Output rate: 50 frames/sec (20 ms stride); no maximum output length

Usage

import json, torch
from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
from huggingface_hub import hf_hub_download
from pyctcdecode import build_ctcdecoder

REPO = "GarchenArchive/wav2vec2-xls-r-300m-rinpoche"
processor = Wav2Vec2Processor.from_pretrained(REPO)
model = Wav2Vec2ForCTC.from_pretrained(REPO).eval()

kenlm_path = hf_hub_download(REPO, "kenlm_5gram.bin")
cfg = json.load(open(hf_hub_download(REPO, "decoder_config.json")))

# Labels straight from the vocab. pyctcdecode wants the CTC blank as "" and the
# word delimiter "|" as " ". The tsheg (་) is an ordinary character -- leave it.
vocab = processor.tokenizer.get_vocab()
labels = [tok for tok, _ in sorted(vocab.items(), key=lambda kv: kv[1])]
labels = ["" if t == "[PAD]" else " " if t == "|" else "⁇" if t == "[UNK]" else t
          for t in labels]

decoder = build_ctcdecoder(labels=labels, kenlm_model_path=kenlm_path,
                           alpha=cfg["alpha"], beta=cfg["beta"])

def transcribe(waveform, sampling_rate=16000):
    inputs = processor(waveform, sampling_rate=sampling_rate, return_tensors="pt")
    with torch.no_grad():
        logits = model(inputs.input_values).logits[0].cpu().numpy()
    return decoder.decode(logits, beam_width=cfg["beam_width"],
                          beam_prune_logp=cfg["beam_prune_logp"],
                          token_min_logp=cfg["token_min_logp"])

For greedy decoding, skip the decoder:

ids = logits.argmax(-1)
text = processor.tokenizer.decode(ids, skip_special_tokens=True)

Do not replace spaces with tshegs. The tokenizer encodes a real space in the transcripts as |, so a space in the output is a genuine space — 94 % of them follow a shad (། ). An earlier version of this card recommended .replace(" ", "་") and a tsheg-as-word-separator decoder alphabet. Both are harmful: on this model's test set they raise SER from 0.3682 to 0.4351 and 0.3840 respectively.

Evaluation

Measured on the test split of GarchenArchive/stt-rinpoche-202609 (1,394 utterances). That split contains the 554 utterances of the earlier stt-rinpoche-202605 test split verbatim, so the middle columns are directly comparable with the previous release.

CER via jiwer; WER and SER (syllable error rate) via tibetan_wer, micro averages.

The KenLM's contribution is small but consistent: SER −0.004, WER −0.003, CER −0.001 against greedy.

Change from the previous release

Same 554 utterances, both decoded with beam + KenLM as above:

The previous release remains available as revision 8a9592c0 of this repo. Note that billingsmoore/garchen-stt is a redirect to this repo, not a separate copy: anything loading that name unpinned now gets these weights. Pin revision="8a9592c0a20fb8dba3ecaa5c6973c67694cb0146" to keep the previous model.

Relative to the encoder-decoder model

Against GarchenArchive/whisper-large-v3-turbo-rinpoche (beam 5), which has a 448-token output ceiling:

This model is stronger on longer clips; whisper is stronger on shorter ones. The two are complementary, and LLM-mediated fusion of their outputs reaches CER 0.1633, WER 0.2804, SER 0.2846 on the full split.

Limitations

The KenLM is stale. It was built on the 202605 transcripts and has not been rebuilt for this release. Rebuilding it on the 202609 transcripts is outstanding.

Substitution-heavy. The model tends to emit near-miss spellings that are close in edit distance without being real words — the complement of the encoder-decoder model's tendency to drop material, which is why fusing the two works.

Vocabulary contains stray symbols. The 129-token vocab includes å, «, ^, ~ and ASCII digits alongside Tibetan. It was not changed for this release, so a few characters in the new material (notably ཿ, ཽ, ཻ) map to [UNK] — about 0.07 % of characters.

Domain. Fine-tuned on a single speaker in a specific register. Prior experiments found that fine-tuning on general Tibetan corpora (Kham, NICTTIB1) degrades performance substantially.

Training

Continued from the previous release on GarchenArchive/stt-rinpoche-202609 (train split: 14,426 utterances, 21.9 h).

The 1e-5 rate is an order of magnitude below the 1e-4 used for fine-tuning from the pretrained base, because this run continues from an already-converged model with a cold optimizer state. Eval CER was still improving when training stopped, so a longer run would likely do better.

Transcriber annotations that were never spoken (❓, parenthesised alternate readings, ༼ ༽) were stripped from the training text.