TransLynx Pakistani Punjabi Whisper (Small)
A fine-tuned version of openai/whisper-small for Pakistani Punjabi speech recognition in Shahmukhi script (the Perso-Arabic script used to write Punjabi in Pakistan).
Model description
This model transcribes spoken Pakistani Punjabi into written Shahmukhi text. It was fine-tuned from OpenAI's multilingual Whisper Small checkpoint, which had a baseline Word Error Rate (WER) of roughly 70% on this language before fine-tuning.
Because Shahmukhi shares its character set with Urdu, the model uses the Urdu language token (ur) internally rather than the Whisper pa token, which corresponds to Eastern Punjabi in Gurmukhi script. The language and task are baked into the generation config, so you do not need to pass them at inference time.
Results
| Evaluation set | WER |
|---|---|
| Baseline (whisper-small, no fine-tuning) | ~70% |
| Validation (seen speakers) | 7.07% |
| Test (completely unseen speaker) | 11.50% |
The test WER is measured on a speaker whose voice was never seen during training, which reflects real-world generalization to new voices.
Intended use
- Transcription of Pakistani Punjabi audio into Shahmukhi text
- Voice interfaces, subtitles, and speech data pipelines for Pakistani Punjabi
- A starting checkpoint for further fine-tuning on additional Punjabi data
How to use
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa
import torch
REPO_ID = "Ghost3454/translynx-pakistani-punjabi-whisper-small"
processor = WhisperProcessor.from_pretrained(REPO_ID)
model = WhisperForConditionalGeneration.from_pretrained(REPO_ID)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()
# load a 16 kHz audio file
audio, _ = librosa.load("sample.wav", sr=16000)
inputs = processor(
audio, sampling_rate=16000, return_tensors="pt"
).input_features.to(device)
with torch.no_grad():
generated_ids = model.generate(inputs)
text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(text)
The language token (ur), task (transcribe), and a repetition guard are already set in the generation config, so no extra generation arguments are required.
Training data
- 3,500 recordings total
- 7 speakers, each recording the same 500 Punjabi sentences
- Sentences written in Shahmukhi script
- One speaker held out entirely as an unseen test set; the remaining six split 90/10 into train and validation
Training procedure
The model was fully fine-tuned (all parameters trained, no LoRA or quantization) on a single consumer GPU.
Key hyperparameters:
- Base model:
openai/whisper-small - Learning rate: 1e-5 with 100 warmup steps
- Effective batch size: 16 (batch size 4 with gradient accumulation of 4)
- Mixed precision: fp16
- Max steps: 2000, evaluating every 200 steps
- Best checkpoint selected by validation WER
- Language token forced to
urfor stable generation no_repeat_ngram_size=3to prevent decoding loops
Limitations
- Trained on a controlled set of 500 sentences across 7 speakers, so vocabulary and acoustic diversity are limited compared to large-scale datasets.
- Performance may drop on spontaneous speech, heavy background noise, dialects far from the training speakers, or vocabulary outside the training sentences.
- Optimized for Shahmukhi output; it is not intended for Gurmukhi (Eastern Punjabi) transcription.
Team & credits
Developed by the TransLynx team:
- Ali Hamza — model fine-tuning, training pipeline, and evaluation
- Farwa — project planning, data collection, and recording
- Jhangir — project planning, data collection, and recording
Thanks to all seven speakers who contributed recordings to the training dataset, without whom this model would not exist.
Citation
If you use this model, please credit the TransLynx Pakistani Punjabi ASR project.