Whisper Medium Hindi-Hinglish ASR (Fine-Tuned)
A fine-tuned version of OpenAI Whisper Medium for Hindi and Hinglish Automatic Speech Recognition (ASR).
This model was developed to improve Whisper's performance on conversational Hindi and code-mixed Hindi-English speech using a carefully curated synthetic speech corpus.
Model Details
| Property | Value |
|---|---|
| Base Model | openai/whisper-medium |
| Model Type | Whisper Encoder-Decoder Transformer |
| Task | Automatic Speech Recognition |
| Languages | Hindi, Hinglish |
| Framework | Hugging Face Transformers |
Motivation
OpenAI Whisper provides excellent multilingual speech recognition, but conversational Hindi and Hinglish often remain challenging due to code-mixing, pronunciation variations, and domain-specific vocabulary.
This project fine-tunes Whisper Medium using a curated synthetic dataset to significantly improve recognition quality while preserving Whisper's multilingual capabilities.
Dataset
The training dataset was generated using a multi-stage synthetic data generation pipeline consisting of:
- LLM-generated conversational Hindi/Hinglish text
- High-quality speech synthesis
- Transcript refinement and normalization
- Quality filtering
- Curated speech-text pairs for Whisper fine-tuning
The focus was on maximizing transcript quality while maintaining linguistic diversity.
Evaluation
Evaluation was performed on an unseen held-out test set.
Overall Performance
| Metric | Score |
|---|---|
| Test Samples | 373 |
| Word Error Rate (WER) | 0.0456 |
| Character Error Rate (CER) | 0.0262 |
| Relative WER Improvement | 86.99% |
Performance by Difficulty
| Difficulty | Samples | WER |
|---|---|---|
| Easy | 317 | 0.0303 |
| Medium | 49 | 0.1301 |
| Hard | 7 | 0.2182 |
Performance by Language
| Language | Samples | WER |
|---|---|---|
| Hindi | 280 | 0.0514 |
| Hinglish | 93 | 0.0300 |
Performance by Error Category
| Category | Samples | WER |
|---|---|---|
| Perfect | 288 | 0.0384 |
| Minor Phonetic | 50 | 0.0628 |
| Digit Format | 29 | 0.0758 |
| Script Switch | 3 | 0.0682 |
| Genuine Error | 2 | 0.0588 |
| Loanword Mishear | 1 | 0.0000 |
Usage
Load the model using the Hugging Face Transformers library.
import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
processor = WhisperProcessor.from_pretrained(
"shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned"
)
model = WhisperForConditionalGeneration.from_pretrained(
"shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned"
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()
Intended Uses
This model is intended for:
- Hindi Automatic Speech Recognition
- Hinglish Automatic Speech Recognition
- Academic Research
- Speech Recognition Experiments
- Fine-tuning Research
Limitations
- The model is primarily trained on synthetic speech.
- Performance may decrease on noisy recordings.
- Performance may vary across unseen accents and speaking styles.
- Not evaluated for streaming ASR.
Training Framework
- Hugging Face Transformers
- PyTorch
Results Summary
| Metric | Value |
|---|---|
| WER | 0.0456 |
| CER | 0.0262 |
| Test Samples | 373 |
| WER Improvement | 86.99% |
Acknowledgements
This project builds upon:
- OpenAI Whisper
- Hugging Face Transformers
Citation
If you use this model in your research or applications, please cite this Hugging Face repository.
@misc{whisper_medium_hindi_hinglish_asr,
title={Whisper Medium Hindi-Hinglish ASR (Fine-Tuned)},
author={Suja Akhter},
year={2026},
publisher={Hugging Face},
howpublished={\url{https://huggingface.co/shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned}}
}