Real-Time Streaming ASR — Kazakh / Russian / English (FastConformer-TDT)
A cache-aware streaming speech-to-text model for Kazakh, Russian, and English,
packaged as a NVIDIA NeMo .nemo checkpoint. Built for real-time / server-side
transcription (WebSocket, low-latency streaming) while staying balanced across all
three languages — no single language is favored.
- Task: automatic speech recognition / speech-to-text (streaming and offline).
- Languages: Kazakh (
kk), Russian (ru), English (en). - Architecture: 115M FastConformer-large (d_model 512, 17 layers, ×8 subsampling),
NeMo
EncDecHybridRNNTCTCBPEModel, TDT decoder (token-and-duration transducer, durations 0–4) + auxiliary CTC (0.3). - Streaming: cache-aware,
att_context_style: chunked_limited, causal convolutions + causal downsampling. Four selectable latency profiles share one set of weights:[[70,13],[70,6],[70,1],[70,0]]≈ 1040 / 480 / 80 / 0 ms right-context lookahead. - Tokenizer: fresh 4096-piece trilingual BPE, balanced (equal lines per language) so Latin/Cyrillic/Kazakh graphemes and digits are all well covered.
- Format: NeMo
.nemo. Input audio: mono 16 kHz.
This is a cache-aware streaming checkpoint (not a full-context offline-only model, not TTS, not diarization).
Evaluation
Held-out WER (%), uniform Whisper text normalization, corpus WER. Evaluation sets: FLEURS test (500 seeded utts / language) for kk/ru/en, and ISSAI KSC2 test (1000 seeded utts, the canonical in-domain Kazakh benchmark). Lower is better. Numbers are measured on these sets, not taken from a model card.
| Decoding profile | Lookahead | KK (FLEURS) | RU (FLEURS) | EN (FLEURS) | KK (KSC2) | FLEURS macro | RTF |
|---|---|---|---|---|---|---|---|
| Offline (full-context attn) | — | 13.50 | 14.95 | 16.31 | 12.69 | 14.92 | ~0.009 |
Streaming [70,13] | ~1040 ms | 13.90 | 14.94 | 16.39 | 12.89 | 15.08 | ~0.015 |
Streaming [70,6] | ~480 ms | 14.34 | 16.20 | 17.34 | 13.75 | 15.96 | ~0.016 |
Streaming [70,1] | ~80 ms | 15.89 | 18.73 | 20.03 | 15.71 | 18.22 | ~0.037 |
Reading the table
- Balanced across languages. FLEURS spread (max−min) is ~2.5–2.9 WER at every profile; RU and EN are not undersampled or sacrificed for KK.
- Real-time at production latency. At
[70,13](~1 s lookahead) the FLEURS macro is 15.08 — matching the offline number, i.e. streaming costs almost nothing at 1 s. RTF ≈ 0.015 (≈ 65× faster than real-time on a single GPU). - Strong on canonical Kazakh. KSC2 is 12.69 offline / 12.89 streaming.
- Graceful degradation all the way down to an 80 ms lookahead for ultra-low-latency use.
Per-profile CER and per-set counts are in metadata/eval.json.
Files
fc_tdt_streaming_kk_ru_en.nemo— NeMo streaming ASR checkpoint.onnx/— end-to-end ONNX export (cache-awareencoder+ TDTdecoder_joint, opset 17) with tokenizer + config for ONNX Runtime / Triton serving. Seeonnx/README.md.metadata/eval.json— full evaluation summary (WER/CER/RTF, all profiles).examples/offline_asr.py— batch/file transcription.examples/streaming_infer.py— cache-aware streaming transcription with a selectable latency profile.
Install
Use an environment with NVIDIA NeMo ASR support and CUDA-enabled PyTorch.
pip install "nemo_toolkit[asr]" huggingface_hub soundfile
Download
from huggingface_hub import hf_hub_download
model_path = hf_hub_download(
repo_id="nur-dev/realtime-streaming-asr-kk-ru-en",
filename="fc_tdt_streaming_kk_ru_en.nemo",
)
Quick start (offline)
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.ASRModel.restore_from(model_path).eval()
print(model.transcribe(["audio_16k_mono.wav"])[0].text)
Streaming latency profile
The model ships four cache-aware profiles in one checkpoint. Select the latency/accuracy trade-off before decoding:
# ~1040 ms lookahead (best accuracy) ... ~80 ms (lowest latency)
model.encoder.set_default_att_context_size([70, 13]) # or [70,6], [70,1]
For true cache-aware streaming inference, use NeMo's
cache_aware_streaming utilities (see examples/streaming_infer.py).
Training
- 8× NVIDIA L40, pure DDP, bf16-mixed, 150k steps.
- ~2413 h trilingual training audio, balanced by hours (kk 33.4 % / ru 33.2 % / en 33.4 %).
- Encoder warm-started from a cache-aware streaming trilingual base; TDT decoder, joint, CTC and the 4096 BPE tokenizer trained fresh for this vocabulary.
- Best blended validation WER: 0.0945.
Intended use & limitations
- Intended for real-time / server-side transcription of kk/ru/en speech at 16 kHz.
- Best accuracy at
[70,13]; use lower profiles when latency matters more than WER. - English at the tightest 80 ms profile (20.0) is the weakest operating point; prefer
[70,13]/[70,6]for English-heavy traffic. - Numbers do not represent written-form/ITN output; RU numerals follow the training convention.
License
Released under CC BY-NC 4.0 (non-commercial). For commercial licensing or inquiries, contact the author.
Contact
Author: Nurgali Kadyrbek — kadyrbeknurgali@gmail.com. For commercial licensing, connect on LinkedIn.