Mega-ASR-MLX-int8
An int8 affine-quantized MLX build of Mega-ASR, derived from
mlx-community/Mega-ASR-MLX-bf16.
Produced for the witness native loader (mlx-mega-asr), which loads the packed
int8 weights directly — no runtime quantization, so the smaller weights are
also the smaller download (~3.8 GB → ~2.2 GB).
Mega-ASR is a robustness layer over Qwen3-ASR-1.7B: a tiny audio-quality router classifies each utterance as clean or degraded and switches a dense LoRA adapter in/out of the base weights at inference.
What is and isn't quantized
The 344 linear projections of the audio encoder + text decoder (q/k/v/o_proj,
fc1/fc2, mlp.{gate,up,down}_proj, conv_out, proj1/proj2, and the tied
embed_tokens) are affine-quantized to int8, group_size 64 — each
<name>.weight is a packed uint32 tensor plus <name>.scales / .biases,
and config.json carries a quantization block.
Everything that must stay precise stays dense bf16: the conv2d subsampling
frontend, all layer norms / biases, the per-head q/k norms, and — critically —
the router and LoRA adapter in extras/. The runtime applies the fp32
LoRA deltas on top of the dequantized base, so the per-utterance router/LoRA
robustness switching is fully preserved. int8 is the deliberate default on
Apple Silicon: batch-1 decode is memory-bandwidth-bound, so int8 is faster
and ~1.8× smaller while staying WER-neutral.
Validation
WER parity vs the bf16 reference is gated by
mlx-mega-asr/examples/int8_prepack_parity.rs (LibriSpeech test-clean): the
pre-packed int8 path must match runtime int8 transcript-for-transcript and stay
within ~0.3% WER of dense bf16. Measured (20 files, Apple Silicon): bf16 1.81% /
int8 1.59% WER, 0/20 transcripts differ from runtime int8, RTF 0.043 → 0.030.
The repo ships vocab.json + merges.txt (the Qwen2 BPE tokenizer is built
from them at load — no tokenizer.json).