Hindi/Hinglish Streaming ASR — Zipformer-M CTC
A from-scratch streaming ASR model for Hindi + code-mixed Hinglish. Architecture: Zipformer-M encoder (~65M) + pure CTC head, causal/chunked for streaming, character-level (Unicode codepoint) vocab. Trained with icefall.
Two releases:
model_p3_epoch15_avg8.pt— CURRENT BEST (Phase 3): scaled up on ~337h.model_p1_epoch30_avg8.pt— Phase 1 baseline (~89h).
Results (CTC greedy, no LM; checkpoint-averaged)
| Eval set | Phase 1 (~89h) | Phase 3 (~337h) |
|---|---|---|
| IndicVoices-hi (Hindi) | 37.2% | 25.7% |
| call-test (internal hinglish calls) | 47.3% | 39.9% |
| Svarah (Indian English) | 50.3% | 44.5% |
Architecture
Zipformer2 "M" (~64.6M): num_encoder_layers=2,2,3,4,3,2, encoder_dim=192,256,384,512,384,256,
downsampling_factor=1,2,4,8,4,2, causal, chunk_size=16,32,64,-1, left_context_frames=64,128,256,-1.
Pure CTC head, 80-dim fbank @16kHz, subsampling 4 (~25 Hz). Vocab 103 tokens
(<blk>,<sos/eos>,<unk>,▁=word-boundary, 63 Devanagari + 26 a-z + 10 digits).
Training data
- Phase 1 (base, from scratch): IndicVoices-hi 60h + keywords 15.4h + Svarah 8.4h + numbers 5.9h (~89h).
- Phase 3 (scale-up, warm-start from P1): IndicVoices-hi 307h + keywords + Svarah + numbers
(~337h base → ~1017h with ×3 speed perturbation).
IndicVoices text uses
unsanitized_normalized(code-switchword [English]→English, tags stripped); all text NFC-normalized, punctuation/foreign scripts dropped, Latin lowercased.
Training
icefall zipformer recipe (char-CTC), PyTorch 2.4.1+cu124 / k2 / lhotse, bf16 (fp16 was unstable
on a single L4), ScaledAdam + Eden. P1: from scratch, base_lr 0.045, 30 ep. P3: warm-start
(--init-ckpt), base_lr 0.02, 15 ep, non-finite-loss batches skipped. Single NVIDIA L4.
Files
model_p3_epoch15_avg8.pt / model_p1_epoch30_avg8.pt — averaged weights + arch config + vocab_size.
tokens.txt — vocab. scripts/ — full reproduction pipeline.
Usage (decode)
Build the icefall zipformer from the checkpoint's config, load_state_dict(ckpt["model"]),
extract 80-dim lhotse fbank, run encoder_embed → encoder → ctc_output → ctc_greedy_search,
map ids via tokens.txt, replace ▁→space. See scripts/eval_wer.py and scripts/server_ctc.py.
Limitations
Greedy CTC, no LM (n-gram/RNN-T would improve WER). English weaker than Hindi (English data-limited).