🏔️ Whisper Small — Balti ASR
State-of-the-art Automatic Speech Recognition model for Balti (بلتی), fine-tuned from openai/whisper-small on Perso-Arabic script audio data.
🏆 Benchmark Summary
- Validation WER: 17.40% (SOTA for Balti ASR)
- Previous Baseline: 26.74% (BaltiVoice) / 22.82% (Wav2Vec2)
👉 For full system details, streaming implementation, and engineering insights, visit the Balti Tarjuman GitHub Repository or read docs/Engineering Case Study.md.
⚡ Quick Start
import torch
import soundfile as sf
from transformers import WhisperForConditionalGeneration, WhisperProcessor
device = "cuda" if torch.cuda.is_available() else "cpu"
repo = "YuvrajGujari/whisper-small-balti"
processor = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(repo).to(device)
audio, _ = sf.read("balti_sample.wav")
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(device)
with torch.no_grad():
predicted_ids = model.generate(inputs, forced_decoder_ids=None)
text = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print("Transcription:", text)
### 📊 Benchmark Comparison
| Model | Setup | Strategy | Validation WER |
| --- | --- | --- | --- |
| **Whisper-small (Champion)** 🏆 | **Cold-Start** | **SpecAugment + $1\times 10^{-4}$ LR** | **17.40%** |
| Wav2Vec2 XLS-R 300M | Cold-Start | Standard CTC | 22.82% |
| *BaltiVoice Literature* | — | Published Baseline | *26.74%* |
| Whisper-small (Round 2) | Warm-Start | Standard Fine-Tuning ($1\times 10^{-5}$ LR) | 36.38% |
| Whisper-small (Base) | Zero-Shot | Out-of-the-Box | 63.42% |
---
### 📈 Training Step Log
| Step | Training Loss | Validation Loss | Validation WER |
| --- | --- | --- | --- |
| 500 | 0.9031 | 0.3995 | 39.72% |
| 1000 | 0.5979 | 0.3019 | 29.23% |
| 1500 | 0.3326 | 0.2518 | 23.81% |
| 2000 | 0.1582 | 0.2235 | 20.27% |
| **2500** | **0.0609** | **0.2142** | **17.40%** |
🛠️ Key Training Highlights
Cold-Start Fine-Tuning: Reset weights to base openai/whisper-small to avoid local minima.
SpecAugment: Applied time and frequency masking (prob=0.05) to prevent overfitting.
LR & Schedule: Trained for 2,500 steps with lr=1e-4 and 500 warmup steps.