Nemotron 3.5 ASR Streaming — Sinhala + Tamil + English (210 h)

A streaming speech recognition model for Sinhala, Tamil and English, fine-tuned from nvidia/nemotron-3.5-asr-streaming-0.6b (cache-aware FastConformer + RNN-Transducer, 0.6 B parameters) on about 210 hours of speech from TriLingDATA/si-ta-en-asr.

• One model for all three languages, with one shared vocabulary (1,019 SentencePiece pieces).
• Works with or without being told the language (target_lang="auto"), which helps with code-switched speech.
• Streaming with selectable look-ahead: 80 ms, 320 ms or 1.12 s.
• Output is lower-case text without punctuation.

Results (validation, 300 clips per language)

Scores are percentages; lower is better. wer_np / cer_np = word / character error rate with punctuation removed from reference and output (references are lower-cased). Validation clips come from speakers that were not used in training.

Final model, by streaming look-ahead (average WER over the three languages):

Validation WER (wer_np) during training — the model improves quickly in the first two epochs and is flat after step 11,000 (changes below about 1 point are within the noise of 300 clips per language):

Raw scores with punctuation (wer / cer in final_model/results.json) are much higher, especially for English (19.1 % raw vs 5.0 % without punctuation), because the references contain punctuation and the model outputs none.

Read CER for Sinhala and Tamil. Their words are long, and many "word errors" are only spacing differences (e.g. இதைப் பற்றி vs இதைப்பற்றி, ගෙදරට ය vs ගෙදරටය) where every letter is correct.

Test-set results (test_si, Common Voice part of test_ta, test_en) will be added here.

Usage

pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA/NeMo.git@v3.0.0" "numba-cuda[cu12]"

from huggingface_hub import hf_hub_download
from nemo.collections.asr.models import ASRModel

path = hf_hub_download("TriLinSL/nemotron-3.5-asr-streaming-si-ta-en-210h", "best_model/model.nemo")
model = ASRModel.restore_from(path).eval()
model.encoder.set_default_att_context_size([56, 13])   # 1.12 s look-ahead; [56, 3] = 320 ms, [56, 0] = 80 ms

# audio: 16 kHz mono WAV/FLAC
print(model.transcribe(["sinhala.wav"], target_lang = "si-LK"))
print(model.transcribe(["tamil.wav"],   target_lang = "ta-IN"))
print(model.transcribe(["english.wav"], target_lang = "en-US"))
print(model.transcribe(["mixed.wav"],   target_lang = "auto"))   # language not given

For true streaming (chunk by chunk with a cache), use NeMo's examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py with this .nemo file.

Training data

• Clips of 1–30 s; speakers of the validation clips were excluded from training.
• Validation: val_si, the Common Voice part of val_ta, val_en (300 clips per language, spread over as many speakers as possible).
• IndicTTS Tamil was not used for training, validation or the tokenizer.
• Labels: Unicode NFC, punctuation removed, Latin letters lower-cased, Sinhala ZWJ/ZWNJ kept.

Training procedure

• Tokenizer: the base tokenizer cannot write Sinhala, so it was replaced by a new SentencePiece unigram vocabulary built as three equal parts (≈ 340 pieces each for Sinhala script, Tamil script and Latin script) merged into one model. The decoder and joint network were re-initialised; the encoder and the language-prompt layer were kept.
• Language prompt: 50 % of training batches used the language slot (si-LK, ta-IN, en-US), 50 % the auto slot.
• Frozen layers: lower 12 of 24 encoder layers frozen (316 M of 623 M parameters trained).
• Optimisation: AdamW, lr 2e-4, 300 warm-up steps, linear decay to 0, weight decay 0.01, gradient clipping 0.5, fp16 mixed precision, 32 clips per step (length-bucketed), 3 epochs = 12,855 steps.
• Streaming: look-aheads [56, 0], [56, 3], [56, 6], [56, 13] sampled during training (as in the base model).
• Hardware: 1 × NVIDIA A40 (48 GB), about 1.5 h of training.

Repository contents

Limitations

• Domains are narrow: read sentences (Sinhala, Tamil) and audiobooks (English). Expect higher error rates on conversational, noisy or telephone speech.
• No punctuation or capitalisation in the output.
• Tamil is trained only on Common Voice read speech; spontaneous Tamil speech is not covered.
• Validation sets are small (300 clips per language); differences below about 1 WER point are within noise.

Data sources and licenses

• Sinhala: SPEAK-ASR/openslr-sinhala-asr (OpenSLR)
• Tamil: Common Voice 22.0 via TriLingDATA/Tamil_STT (CC0)
• English: mythicinfinity/libritts_r (LibriTTS-R)

The model weights are derived from nvidia/nemotron-3.5-asr-streaming-0.6b (© NVIDIA Corporation) and are released under the same license as the base model (see license above and the base model card). Use of this model must also follow the license of each data source. Please cite the base model and the source datasets.