Whisper Small — Gujarati + Talpada (તળપદા)
A fine-tuned Whisper Small ASR model for Gujarati (gu), trained so it also recognizes Talpada — the regional/village dialect of Gujarati — which vanilla Whisper frequently misses.
Model is saved in fp16 (~484 MB).
How it was trained
Base model: openai/whisper-small (244M params)
Dataset: shunyalabs/gujarati-speech-dataset — a large collection of Gujarati speech + transcripts, streamed from Hugging Face as parquet shards.
- Data: 5 parquet shards, ~2,000 samples (10-second audio windows)
- Eval: 30 held-out samples
Fine-tuning strategy (memory-efficient, runs on 16 GB GPU / Colab free tier):
- Encoder: fully frozen
- Decoder: only self-attention, layer-norms, and token embeddings trained (~65M of 244M params); cross-attention and FFN frozen → tiny optimizer, low VRAM
- Batch size 2 × gradient accumulation 8 (effective 16), learning rate 5e-5, warmup 100 steps, 3 epochs (372 steps), fp16, gradient checkpointing
- Transformers 4.57.6
Result: training loss dropped 1.65 → 0.232.
Performance
Measured on 30 held-out samples from the same dataset:
| Metric | Value |
|---|---|
| Training loss (final) | 0.232 |
| Held-out WER | 72.37% |
⚠️ Read the WER carefully: during eval, input audio was truncated to 10 seconds while the reference transcripts were the full sentences (up to ~25s of speech). The model could not transcribe words it never heard, which inflates WER substantially. Qualitatively the model produces phonetically accurate Gujarati — most errors are minor spelling variants (e.g. ક્રાઉન ઑફિસ → ક્રાઉન ઑફેસ), not missed words.
Comparison to other models: no strict apples-to-apples baseline was run on the same held-out set. Gujarati is a weak language for vanilla Whisper, and this fine-tune is a domain adaptation for Gujarati/Talpada speech, not a state-of-the-art benchmark model. Expect it to clearly beat stock whisper-small on spoken Gujarati, while large models (whisper-large-v3) will still win on formal/news audio — this model's edge is dialect words and low-resource robustness.
Limitations
- Trained on a small subset (~2,000 samples, 3 epochs) — coverage of rare Talpada words is limited to the training data
- Not trained on custom Talpada audio yet; the dialect vocabulary is learned from standard Gujarati text only
- Performs best on short phrases and conversational speech; dense news sentences with English loanwords remain hard
- fp16 weights: load on GPU/MPS, or convert to fp32 for CPU inference
Usage
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch
processor = WhisperProcessor.from_pretrained("princetunes/whisper-small-gujarati-talpada")
model = WhisperForConditionalGeneration.from_pretrained("princetunes/whisper-small-gujarati-talpada")
model.to("cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu"))
model.eval()
import librosa
audio, sr = librosa.load("your_audio.wav", sr=16000, mono=True)
feats = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(model.device)
forced = processor.get_decoder_prompt_ids(language="gujarati", task="transcribe")
with torch.no_grad():
ids = model.generate(input_features=feats, forced_decoder_ids=forced, max_length=225)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])
Training code
Training was done in a Colab notebook (streamed data loading, frozen-encoder decoder fine-tune). See the shunyalabs/gujarati-speech-dataset repo for the raw data; contact the model author for the notebook.