Nemotron 3.5 ASR Streaming 0.6B — ONNX INT4 k-quant (CPU, opset 24, chunk 1.12s)
INT4 k-quant (MatMulNBits) — smallest and fastest ONNX model converted from NVIDIA Nemotron 3.5 ASR Streaming 0.6B to native ONNX opset 24 for CPU inference with ONNX Runtime GenAI.
Streaming chunk size: 1.12s — maximum encoder context — recommended for INT4 (extra context compensates quantization noise).
Model Details
| Property | Value |
|---|---|
| Base model | NVIDIA Nemotron 3.5 ASR Streaming 0.6B |
| ONNX opset | 24 (native) |
| Encoder layers | 24 |
| Hidden size | 1024 |
| Precision | INT4 k-quant |
| Chunk size | 1.12s (17920 samples @ 16kHz) |
| Left context | 140 encoded frames |
| Sample rate | 16,000 Hz |
| Vocab size | 13,088 |
| Languages | 80+ (multilingual) |
| VAD | Silero VAD included |
| Total size | ~793 MB |
Files
| File | Size |
|---|---|
encoder.onnx + .data | 682 MB |
decoder.onnx + .data | 57 MB |
joint.onnx + .data | 36 MB |
silero_vad.onnx | 2 MB |
Requirements
- ONNX Runtime ≥ 1.25 (opset 24 support on CPU).
- Tested with
Microsoft.ML.OnnxRuntime 1.25.1+Microsoft.ML.OnnxRuntimeGenAI 0.14.1.
Quick Start
C# / .NET
using NemotronSpeech;
using var session = new ModelSession(
modelPath: "path/to/this/model",
executionProvider: "cpu",
langId: null,
useVad: false);
CLI
NemotronSpeech.exe <model_path> <audio_file> cpu
NemotronSpeech.exe <model_path> --mic cpu --language en
Choosing a chunk size
| Chunk | Latency | Encoder context | Best for |
|---|---|---|---|
| 0.08s / 0.16s | Lowest | Minimal | Ultra-low-latency dictation |
| 0.56s | Balanced | 70 frames | Recommended default (NVIDIA) |
| 1.12s | Higher | 140 frames | Best accuracy, code-switching, multilingual |
Larger windows give the encoder more context, which improves WER — especially noticeable for quantized (INT4) builds and code-switched / multilingual audio.
Related models
DimQ1/nemotron-3.5-asr-streaming-0.6b-onnx-fp32-cpu— opset 21 FP32DimQ1/nemotron-3.5-asr-streaming-0.6b-onnx-int4-cpu— opset 21 INT4 (fastest)