whisper-large-v3-urdu-v2
Whisper-large-v3 fine-tuned for Pakistani Urdu in its three written registers, selected at decode time by a control token:
| token | register | output looks like |
|---|---|---|
| `< | scripturdu | >` |
| `< | scriptroman | >` |
| `< | scriptmixed | >` |
The mixed register keeps English islands in Latin letters inside an Urdu-script matrix — the convention of real Pakistani speech (dramas, call-centres, interviews) — including English function words and phrases, not just borrowed nouns.
Usage
The script token goes after <|notimestamps|> in the decoder prefix:
import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
repo = "emads/whisper-large-v3-urdu-v2"
processor = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(repo).eval()
def prefix(script_token: str) -> torch.Tensor:
toks = ["<|startoftranscript|>", "<|ur|>", "<|transcribe|>",
"<|notimestamps|>", script_token]
return torch.tensor([processor.tokenizer.convert_tokens_to_ids(toks)])
feats = processor(audio_16khz, sampling_rate=16000, return_tensors="pt").input_features
out = model.generate(feats, decoder_input_ids=prefix("<|scripturdu|>"), max_new_tokens=200)
print(processor.batch_decode(out, skip_special_tokens=True)[0])
Swap "<|scripturdu|>" for "<|scriptroman|>" or "<|scriptmixed|>" to
change the output convention on the same audio.
Evaluation (2026-08-25)
Greedy decoding, register-matched control token, text normalization as in the
training repo (github.com/emad-siddiq/urdu-llm, urduwhisper/text_norm.py).
| benchmark | metric | this model | whisper-large-v3 fine-tuned v1 |
|---|---|---|---|
| FLEURS ur_pk test (n=299) | WER | 0.1665 | 0.1687 |
| Common Voice Urdu test (n=1000) | WER | 0.1598 | 0.1574 |
| Roman-Urdu test, 300 clips | CER | 0.1470 | 0.857 (v1 cannot write Roman) |
| Urdu–English code-switch, 1000 clips | WER | 0.3540 | 0.3581 (v1 writes islands in Urdu script) |
On the code-switch set, 37% of English islands are written in Latin (v1: 0%) with 80% heard in some script; script fidelity is 1.000 for the Urdu register and 0.981 for Roman — the registers do not leak into each other.
Provenance notes: the Roman references are LLM-generated chat-Roman, mechanically audited (no content errors in 300/300; native naturalness review pending). The code-switch benchmark is the manually-corrected gold slice of UrduSpeech (CC-BY-4.0, arXiv:2605.17846).
Training
Fine-tuned from the v1 Urdu fine-tune of openai/whisper-large-v3 (2,500
steps, batch 32, lr 5e-6) on a register-tagged mix: Urdu-script speech
(Common Voice, IndicVoices, Kathbath, PRUS, transliterated IndicVoices-Hindi),
Roman-Urdu targets (same audio, romanized labels), and a mixed bucket under
<|scriptmixed|>: MUCS Hinglish (re-scripted to the Urdu convention), English
audio with Latin labels (FLEURS-en, LibriSpeech clean-100), on-the-fly spliced
Urdu+English utterances, and paired loanword copies. Latin-letter label tokens
of mixed rows carry 2× loss weight. All sources permissively licensed
(CC-0/CC-BY); the benchmark corpora never enter training (leakage-filtered).
Limitations
- Long-form audio needs external chunking (30 s window).
- Code-switch WER is dominated by conversational-domain recognition, not script choice; Pakistani conversational audio is the known gap.
- Roman spelling follows chat conventions; expect variant spellings (ke/kay, hai/he) — score with variant folding.
- Decoding without a script token is unsupported; always pass the prefix.