parakeet-tdt-0.6b-v2-mlx-8bit
8-bit quantized MLX weights for NVIDIA Parakeet TDT 0.6B v2, ready to load on Apple Silicon without any quantization step at runtime.
Published by CosmicFusion Labs for WhisperBud, an on-device dictation app for macOS.
Why this exists
The upstream MLX weights (mlx-community/parakeet-tdt-0.6b-v2) are
full precision: 2,472 MB to download, then quantized on each machine at
load time. These weights are the result of that same quantization
(mlx.nn.quantize, bits=8, group_size=64), saved once:
| upstream | this repo | |
|---|---|---|
| download | 2,472 MB | 933 MB |
| model load | ~1.7 s | ~0.05 s |
| transcription output | reference | identical |
Verified byte-identical transcripts against the quantize-at-load path on real speech clips (English, 5 to 45 seconds).
Usage
parakeet_mlx.from_pretrained() does NOT load quantized weights: it builds a
full-precision model and loads the tensors into it. Build the quantized
structure first, then load:
import json
import mlx.nn as nn
from huggingface_hub import hf_hub_download
from parakeet_mlx.utils import from_config
repo = "cosmicfusionlabs/parakeet-tdt-0.6b-v2-mlx-8bit"
config = json.load(open(hf_hub_download(repo, "config.json")))
quant = config.pop("quantization") # {"bits": 8, "group_size": 64}
model = from_config(config)
nn.quantize(model, bits=quant["bits"], group_size=quant["group_size"])
model.load_weights(hf_hub_download(repo, "model.safetensors"))
model.eval()
result = model.transcribe("audio.wav")
print(result.text)
Accuracy
Same weights as quantizing the upstream model at load time, so accuracy is
unchanged from the 8-bit path. In our own benchmark on real (non-synthetic)
English speech, 8-bit Parakeet TDT 0.6B v2 measured 11.6% mean WER, between
whisper-small.en (13.5%) and whisper-medium.en (8.2%), while running
2 to 4x faster than medium.en on an M1 Pro. 4-bit was measured too and
degraded accuracy (13.3% WER), which is why this repo ships 8-bit.
Credits and license
- Original model: nvidia/parakeet-tdt-0.6b-v2 (CC-BY-4.0)
- MLX conversion: mlx-community/parakeet-tdt-0.6b-v2
- Runtime: parakeet-mlx
Released under CC-BY-4.0, the same license as the original NVIDIA model.