Whisper Small — Sinhala (experiments)

Experimental fine-tuned variants of openai/whisper-small for Sinhala automatic speech recognition (ASR). The best model, run11 (16.38% test WER), has its own repo: SinhaSpeech/whisper-small-sinhala-v6-e6-run11-best. All training data lives in SinhaSpeech/sinhala-asr-data.

This repo bundles ten training runs (run1 to run10) so the experiments are reproducible from one place.

Repo layout

models/
  run1/   full fine-tune, v1 data — merged, ready-to-use model
  run2/   LoRA (AMD hardware), v1 data — merged, ready-to-use model
  run3/   LoRA adapter only, v1 data — best checkpoint (epoch 2), run interrupted at epoch 2.5
  run4/   LoRA adapter only, v1 data — r=32, six target modules
  run5/   full fine-tune, v4 data (speaker-disjoint, spacing-normalised)
  run6/   full fine-tune, v5 data, lr 3e-5 linear, 5 epochs   <- best v5 model
  run7/   as run6 but cosine scheduler
  run8/   as run6 but 4 epochs
  run9/   as run6 but lr 2e-5 and 4 epochs
  run10/  full fine-tune, v6 data, lr 3e-5 linear, 5 epochs (also SinhaSpeech/whisper-small-sinhala-v6-e5)
checkpoints/
  run5/   resume notes and training log for run5

All splits (v1 to v6) are in SinhaSpeech/sinhala-asr-data under data/stratified_v1 to data/stratified_v6. The v1 to v4 copies that used to be bundled here were identical and have been removed.

Results

Test WER / CER in percent. run1 to run4 were scored on the v1 test set (15,483 utterances), which is not speaker-disjoint, so its numbers are optimistic. run5 to run10 were scored on the speaker-disjoint test set (15,860 utterances); v5 also normalises about 8.5% of the labels, so run5 vs run6 is indicative only. Text is lower-cased with punctuation removed before scoring.

Which run should I use?

Usage

Full / merged models (run1, run2, run5 to run10)

from transformers import WhisperForConditionalGeneration, WhisperProcessor

repo = "SinhaSpeech/whisper-small-sinhala-experiments"
model = WhisperForConditionalGeneration.from_pretrained(repo, subfolder="models/run6")
processor = WhisperProcessor.from_pretrained(repo, subfolder="models/run6")

LoRA adapters (run3, run4)

from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel
from huggingface_hub import snapshot_download

base = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
path = snapshot_download("SinhaSpeech/whisper-small-sinhala-experiments", allow_patterns="models/run3/*")
model = PeftModel.from_pretrained(base, f"{path}/models/run3")
processor = WhisperProcessor.from_pretrained(f"{path}/models/run3")

Training data

All runs were trained on Sinhala speech aggregated from OpenSLR-52, YouTube, BizBrains, and Linga sources (~154,828 examples total). See the dataset card for split details and column schema.

Limitations

• stratified (v1) and stratified_v2 may share speakers between train and eval splits; use stratified_v3 or later for a speaker-independent evaluation.
• Errors that remain in the best model are mostly spelling conventions (spoken vs written forms), rare words and long utterances; see the project's error analysis.
• The LoRA runs can repeat words in long outputs; no_repeat_ngram_size=3 is a recommended decoding setting for them.
• English forgetting was measured only for some early runs; it was not evaluated for run5 to run10.