moona3k/mlx-qwen3-asr-0.6b-8bit
8-bit (group size 64) MLX quantization of Qwen/Qwen3-ASR-0.6B for mlx-qwen3-asr, a ground-up MLX reimplementation of Qwen3-ASR for Apple Silicon. No PyTorch, no transformers, no model conversion step on the user's side.
Usage
pip install -U mlx-qwen3-asr
mlx-qwen3-asr audio.wav --model moona3k/mlx-qwen3-asr-0.6b-8bit
import mlx_qwen3_asr as m
result = m.transcribe("audio.wav", model="moona3k/mlx-qwen3-asr-0.6b-8bit")
print(result.text)
Requires mlx-qwen3-asr >= 0.4.1. Word timestamps (--timestamps) and the
HTTP server (mlx-qwen3-asr serve --model moona3k/mlx-qwen3-asr-0.6b-8bit) work unchanged.
What is quantized
- Every
LinearandEmbeddinglayer in the text decoder and the audio encoder is affine-quantized to 8 bits with group size 64 (mlx.nn.quantize). - Remaining floating tensors (scales, biases, norms, conv stem) are stored in float16, so inference runs in float16 end to end.
lm_headis tied to the token embedding in the source model and is not stored twice; the loader re-ties it.- Download size: 801M (fp16 source: 1.8 GB).
Quality
LibriSpeech test-clean, 100 speaker-balanced clips (speaker_round_robin),
greedy decoding, Apple M4 Pro, MLX 0.30.6:
| Model | WER | CER |
|---|---|---|
Qwen/Qwen3-ASR-0.6B fp16 | 2.33% | 0.59% |
| this artifact (8-bit g64) | 2.33% | 0.59% |
Hypotheses: identical to fp16 on all 100 clips.
Latency envelope from the committed quantization matrix
(docs/benchmarks/2026-09-07-quant-matrix-test-clean-speaker100.md, 0.6B):
8-bit runs about 2.4x and 4-bit about 2.7x faster than fp16 on a 10 s clip.
Per-sample JSON for this artifact's evaluation is committed in the
mlx-qwen3-asr repository as
docs/benchmarks/2026-09-19-quantized-artifacts-librispeech-test-clean-100-0.6B_8bit.json.
Reproduce
git clone https://github.com/moona3k/mlx-qwen3-asr && cd mlx-qwen3-asr && git checkout v0.4.3
python scripts/convert.py --model Qwen/Qwen3-ASR-0.6B --quantize 8 --encoder-bits 8 \
--group-size 64 --dtype float16 --output-dir Qwen3-ASR-0.6B-8bit-g64
python scripts/eval_librispeech.py --model Qwen3-ASR-0.6B-8bit-g64 --samples 100 --sampling speaker_round_robin
License
Apache-2.0, following the source model.