Nemotron 3.5 ASR Streaming 0.6B — ONNX INT8 (CPU, 0.56s chunks)

INT8 k-quant quantized ONNX model converted from NVIDIA Nemotron 3.5 ASR Streaming 0.6B using Microsoft Olive.

Optimized for fast CPU inference with ONNX Runtime GenAI. Uses NVIDIA's recommended 0.56s chunk size for best latency/accuracy trade-off.

Model Details

Quick Start

C# / .NET (onnxruntime-genai)

using NemotronSpeech;

using var session = new ModelSession(
    modelPath: "DimQ1/nemotron-3.5-asr-streaming-0.6b-onnx-int8-cpu",
    executionProvider: "cpu",
    langId: "11",  // Russian
    useVad: true
);

Python (onnxruntime-genai)

import onnxruntime_genai as og
import numpy as np

model = og.Model("DimQ1/nemotron-3.5-asr-streaming-0.6b-onnx-int8-cpu")
processor = og.StreamingProcessor(model)
tokenizer = og.Tokenizer(model)
params = og.GeneratorParams(model)
generator = og.Generator(model, params)

generator.set_runtime_option("lang_id", "11")  # Russian

chunk = np.zeros(8960, dtype=np.float32)  # 0.56s @ 16kHz
inputs = processor.process(chunk)
if inputs is not None:
    generator.set_inputs(inputs)
    while not generator.is_done():
        generator.generate_next_token()

Conversion Recipe

cd converter
python src/optimize.py \
  --model-name nemotron-3.5-asr-streaming-0.6b.nemo \
  --encoder-precision int8 \
  --execution-provider cpu \
  --output-dir build/onnx_models_int8_cpu

See nemotron-speech-csharp for the full C# demo app with VoiceType WPF UI.

Files

Available Variants

License

This model inherits cc-by-nc-4.0 from the base NVIDIA Nemotron model.