Whisper Small — Bengali ASR (fine-tuned)
This model is a fine-tuned version of Noobbbbb/whisper-small-bn
on the Noobbbbb/bengali-ai-asr-80k dataset,
for automatic speech recognition (ASR) in Bengali.
Model Details
- Base model:
Noobbbbb/whisper-small-bn(Whisper-small architecture, previously fine-tuned for Bengali) - Architecture: Whisper small (12 encoder layers, 12 decoder layers, 768 hidden size)
- Language: Bengali (
bn) - Task:
transcribe - Fine-tuning framework: 🤗
transformersSeq2SeqTrainer
Training Data
- Dataset:
Noobbbbb/bengali-ai-asr-80k - Train subset used: first 40,000 examples of the
trainsplit - Validation subset used: first 4,000 examples of the
valsplit - Audio was processed via
WhisperProcessor(log-mel spectrogram input features); transcripts were tokenized with the Whisper tokenizer.
Training Procedure
| Hyperparameter | Value |
|---|---|
| Epochs | 3 |
| Per-device train batch size | 64 |
| Per-device eval batch size | 32 |
| Gradient accumulation steps | 1 |
| Learning rate | 1e-5 |
| Warmup steps | 50 |
| Precision | fp16 (mixed precision) |
| Gradient checkpointing | enabled |
| Optimizer | adamw_torch_fused |
| Generation max length | 225 |
| Eval strategy | every 400 steps |
Training Results — Stage 1 (initial fine-tune)
- Final training loss: 0.2969
- Total steps: 1,875 (3 epochs)
- Train runtime: ~6,500s (~108 minutes)
- Throughput: ~18.5 samples/sec
Evaluation used Word Error Rate (WER) and Character Error Rate (CER) via the evaluate library
(wer_metric / cer_metric), computed every 400 steps on the 4,000-example validation subset.
| Step | Training Loss | Validation Loss | WER | CER |
|---|---|---|---|---|
| 400 | 0.3408 | 0.1606 | 0.5021 | 0.1601 |
| 800 | 0.2654 | 0.1340 | 0.4438 | 0.1360 |
| 1200 | 0.2472 | 0.1204 | 0.4193 | 0.1273 |
| 1600 | 0.2179 | 0.1159 | 0.4101 | 0.1261 |
| 1875 (final) | 0.2159 | 0.1144 | 0.4075 | 0.1262 |
Stage 1 final validation WER: 40.75% · CER: 12.62%
Training Results — Stage 2 (extended fine-tune, +5 epochs)
Training was continued for 5 additional epochs from the Stage 1 checkpoint.
- Final training loss: 0.1609
- Total steps: 6,645 (5 epochs this stage)
- Train runtime: ~17,215s (~4.78 hours)
- Throughput: ~24.7 samples/sec
Stage 2 out-of-domain evaluation:
Evaluated on google/fleurs (bn_in config)
| Metric | Value |
|---|---|
| WER | 38.90% |
| CER | 13.98% |
This was run on an out-of-domain evaluation set (i.e. not the in-domain validation subset used for Stage 1 tracking). Compared to Stage 1's in-domain validation numbers, WER improved (40.75% → 38.90%) while CER is slightly higher (12.62% → 13.98%) — consistent with evaluating on different, harder data rather than a straightforward regression.
Sample Predictions (Stage 2)
| Prediction | Reference |
|---|---|
| একটি দল কৃষি যন্ত্রপাত্ত্তী তৈরী করে এবং অন্য দলটি, আলংকারিক এবং ট্রুটেমিও স্তম্ভ তৈরী করে। | একটি দল কৃষি যন্ত্রপাতি তৈরি করে এবং অন্য দলটি আলংকারিক এবং টোটেমীয় স্তম্ভ তৈরি করে। |
| জবরদস্থক খান বিদ্রোহীকে পরাভূত করেন। | জবরদস্ত খান বিদ্রোহীকে পরাভূত করেন। |
| তিনিই বামলা চলচ্চিত্র অবশেষে তার কণ্ঠস্বর প্রদানের জন্য জাতীয় চলচ্চিত্র পুরস্কার জিতে নেন। | তিনি বাংলা চলচ্চিত্র "অবশেষে" তার কণ্ঠস্বর প্রদানের জন্য জাতীয় চলচ্চিত্র পুরস্কার জিতে নেন। |
Qualitatively, Stage 2 predictions track the reference more closely than Stage 1 (fewer inserted/garbled syllables), though small substitution and diacritic errors remain.
How to Use
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch, librosa
repo_id = "Noobbbbb/bengali-ai-asr-80k" # update if you rename the repo
processor = WhisperProcessor.from_pretrained(repo_id)
model = WhisperForConditionalGeneration.from_pretrained(repo_id)
model.generation_config.language = "bengali"
model.generation_config.task = "transcribe"
waveform, sr = librosa.load("audio.wav", sr=16000, mono=True)
inputs = processor(waveform, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
predicted_ids = model.generate(inputs["input_features"])
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)
For longer audio
from transformers import pipeline, WhisperForConditionalGeneration, WhisperProcessor, AutoModelForSpeechSeq2Seq
model_id = "Noobbbbb/whisper-small-bn-v2"
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id)
if isinstance(model.generation_config.eos_token_id, list):
model.generation_config.eos_token_id = model.generation_config.eos_token_id[0]
if isinstance(model.config.eos_token_id, list):
model.config.eos_token_id = model.config.eos_token_id[0]
processor = WhisperProcessor.from_pretrained(model_id)
waveform, _ = librosa.load("/content/yt.mp3")
pipe = pipeline(
"automatic-speech-recognition",
model=model,
tokenizer=processor.tokenizer,
feature_extractor=processor.feature_extractor,
device=0 if torch.cuda.is_available() else -1,
)
result = pipe(
waveform,
return_timestamps=True, # enables long-form transcription
generate_kwargs={"language": "bengali"}
)
Note: This will return timestamp-wise transcription
Limitations
- Stage 1 was fine-tuned only on a 40k-example subset of the full dataset; Stage 2's exact train subset size is unconfirmed (see note above).
- Stage 1 metrics (WER 40.75%, CER 12.62%) are in-domain validation numbers; Stage 2 metrics (WER 38.90%, CER 13.98%) are from an out-of-domain evaluation set, so the two aren't directly comparable apples-to-apples.
- Performance on noisy/telephony-style audio (e.g. call center recordings) has not been separately validated for this checkpoint.
Citation / Acknowledgements
- Base model:
Noobbbbb/whisper-small-bn - Dataset:
Noobbbbb/bengali-ai-asr-80k - Built on OpenAI's Whisper architecture via 🤗 Transformers.