Whisper Small — Santali ASR (Custom Transliteration / Sanlish)
Fine-tuned version of openai/whisper-small for Automatic Speech Recognition (ASR) of the Santali language, using a custom Roman-script transliteration scheme called Sanlish.
This model is part of a thesis research pipeline for low-resource Santali speech processing, targeting the task of Santali Speech → Sanlish Transcription → Bangla Translation.
Model Details
Model Description
- Developed by: thunderboltc
- Model type: Whisper Small (Encoder-Decoder Transformer, Seq2Seq)
- Language: Santali (
sat) — transcribed in custom Roman transliteration (Sanlish) - Base model: openai/whisper-small
- Parameters: 241,734,912 (~242M)
- Fine-tuned for: Low-resource ASR — Santali speech to Sanlish text
- License: MIT
What is Sanlish?
Sanlish is a custom Roman-script transliteration of Santali designed to bridge the gap between Santali phonetics and Latin-alphabet-based ASR systems. It was developed as part of this thesis to serve as an intermediate representation between Santali audio and Bangla translation.
Training Details
Training Data
- Dataset: 502 Santali audio recordings with parallel Sanlish transliteration annotations
- Collected by: thunderboltc (thesis dataset, not yet publicly released)
- Split:
| Split | Size | Percentage |
|---|---|---|
| Train | 401 | 80% |
| Validation | 50 | 10% |
| Test | 51 | 10% |
- Audio format: WAV, 16kHz mono
- Text target: Custom Roman transliteration (Sanlish)
Training Procedure
Preprocessing
- Audio resampled to 16,000 Hz
- Log-mel spectrogram features extracted using WhisperFeatureExtractor (80 mel bins, 3000 frames)
- Text tokenized using WhisperTokenizer with
language=english, task=transcribe forced_decoder_idsandsuppress_tokensdisabled to allow custom vocabulary transcription
Training Hyperparameters
| Parameter | Value |
|---|---|
| Epochs | 25 |
| Learning Rate | 1e-5 |
| Batch Size (per device) | 8 |
| Gradient Accumulation Steps | 2 (effective batch = 16) |
| Warmup Steps | 50 |
| Precision | fp16 mixed precision |
| Evaluation Strategy | Per epoch |
| Best Model Selection | Lowest validation WER |
| Hardware | NVIDIA Tesla T4 (Google Colab) |
| Training Duration | ~73 minutes |
Best Checkpoint
Epoch 13 was selected as the best checkpoint based on lowest validation WER (53.41%).
Evaluation
Results on Test Set (51 utterances)
| Metric | Value |
|---|---|
| Word Error Rate (WER) | 56.55% |
| Character Error Rate (CER) | 13.31% |
These results are consistent with expectations for a low-resource ASR system trained on ~400 utterances. The large gap between WER and CER reflects that the model captures phonetic character structure well, even when full word sequences deviate from the reference.
Sample Predictions
| Reference | Prediction |
|---|---|
| onde alom chala a | onde alom chala a ✓ |
| tehindo adi raashkaien yaikaoda | tehindo adi reskaien yaikaoda |
| gapa shetah ale saote baskiyago meh | gapa setah ale saote baskiyago meh |
| shetay bhuh da | shetay bhuheda |
How to Get Started
import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa
# Load model and processor
processor = WhisperProcessor.from_pretrained("thunderboltc/whisper-small-santali-sanlish")
model = WhisperForConditionalGeneration.from_pretrained("thunderboltc/whisper-small-santali-sanlish")
# Load audio (must be 16kHz)
audio, sr = librosa.load("your_santali_audio.wav", sr=16000)
# Preprocess
inputs = processor.feature_extractor(audio, sampling_rate=16000, return_tensors="pt")
# Generate transcription
with torch.no_grad():
predicted_ids = model.generate(inputs.input_features)
# Decode
transcription = processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)
print("Sanlish transcription:", transcription[0])
Intended Use
Direct Use
Transcribing Santali speech into Sanlish (custom Roman transliteration) for downstream NLP tasks.
Downstream Use
This model is designed to feed into a Machine Translation (MT) model (NLLB-200) for end-to-end Santali Speech → Bangla Text translation.
Out-of-Scope Use
- Not suitable for general-purpose ASR in other languages
- Not recommended for standard Santali Ol Chiki script transcription
- Performance may degrade on speaker accents or recording conditions not represented in the 502-utterance training set
Limitations & Bias
- Trained on only 502 utterances — a very small dataset by ASR standards
- May not generalize well to unseen speakers or recording environments
- Sanlish transliteration scheme has inherent inconsistencies (word boundary variation, hyphenation) which inflate WER beyond true phonetic error rate
- No speaker diversity information available for the training set
Citation
If you use this model in your research, please cite:
@misc{thunderboltc2025whisper-santali,
author = {thunderboltc},
title = {Whisper Small Fine-tuned on Santali Custom Transliteration (Sanlish)},
year = {2025},
publisher = {Hugging Face},
url = {https://huggingface.co/thunderboltc/whisper-small-santali-sanlish}
}
Environmental Impact
- Hardware: NVIDIA Tesla T4
- Cloud Provider: Google Colab
- Hours used: ~1.25 hours
- Carbon emissions: Estimated using the ML Impact Calculator