Nemotron-3.5 ASR Streaming 0.6B — MLX INT8

Cache-aware multilingual Conformer + RNN-T speech recognition from NVIDIA, converted to native MLX for streaming inference on Apple silicon. This bundle uses affine group-64 8-bit linear weights, BF16 recurrent state, and unquantized convolution, embedding, LSTM, and language-prompt layers.

Model

Files

Streaming accuracy

FLEURS test, 50 samples per language, cache-aware 320 ms streaming. English uses Whisper's EnglishTextNormalizer; German, French, and Arabic use BasicTextNormalizer; Hindi and Japanese use BasicTextNormalizer(split_letters=True).

Mean WER across these six slices is 11.01% and mean CER is 5.50%.

Streaming performance

M5 Pro with 48 GB unified memory, one fresh Python process, 63.7 seconds of concatenated FLEURS English audio, 199 × 320 ms chunks:

The post-load value reflects MLX's lazy memory mapping; peak RSS is the useful resident-memory figure after inference touches the weights.

Swift usage

The native runtime is in NemotronStreamingASR.

import NemotronStreamingASR

let model = try await NemotronStreamingASRMLXModel.fromPretrained(
    variant: .int8
)
let session = try model.createSession(language: "en-US")

for pcm320ms in audioChunks {
    for partial in try session.pushAudio(pcm320ms) {
        print(partial.text)
    }
}

for final in try session.finalize() {
    print(final.text)
}

Sessions retain the mel, attention, convolution, and RNN-T predictor caches. Model calls are serialized so multiple source-local sessions can safely share one loaded weight set.

Python reference runtime

The reproducible exporter, cache-aware runtime, tests, and benchmarks live in speech-models/models/nemotron-asr-streaming-multilingual/export.

hf download aufklarer/Nemotron-3.5-ASR-Streaming-0.6B-MLX-8bit --local-dir ./nemotron-mlx

python streaming.py \
  --bundle ./nemotron-mlx \
  --wav recording.wav \
  --lang en-US \
  --chunk-ms 320 \
  --print-partials

--print-partials prints cumulative text whenever a chunk adds visible tokens. Both reference runtimes remove angle-bracket language markers such as <en-US> from user-facing text.

Choosing a precision

INT5 minimizes resident memory; INT8 preserves more quantization headroom. Benchmark both on domain audio before deployment.

Limitations

• This is ASR only. It does not perform speaker diarization or speaker identification.
• Language conditioning is explicit. Pass a supported BCP-47 language tag or use the bundle's auto slot.
• Partial text is provisional and may end mid-word. Applications should replace the previous partial until the caller finalizes the utterance.
• The native MLX path does not currently implement the Core ML runtime's optional word-boosting layer.
• The runtime is intended for Apple silicon and requires the MLX Metal shader library to be packaged with the executable.

Provenance

License

Distributed under OpenMDW 1.1, matching the upstream NVIDIA model. Review the upstream model card and license before deployment.

Links

• speech-swift
• speech-models
• Soniqo documentation
• NVIDIA source model