Qwen3.8-Flash-Next-Uncensored — MLX mixed-precision 4.86 bpw

Mixed-precision AWQ/affine quantization of orcarouter/Qwen3.8-Flash-Next-Uncensored (qwen4_exp, 180B) written to fit an M5 Max 128 GB box at full 262k context.

109.24 GB · BPW 4.86 · 180.00B params · 3.30× vs bf16 (360.00 GB) — routed experts 4-bit gs64, n-gram PLE table 4-bit gs32, GDN in_proj/out_proj 8-bit gs64, attention q/k/v + embeddings + lm_head + shared experts + vision 8-bit, o_proj 6-bit gs64, the sparse-attention indexer exact bf16, routers / hyper-connections / norms / SSM state exact bf16.

How it was made

• Hand-written per-module {bits, group_size} map (mode: affine), applied by monkey-patching omlx.oq.universal_quant_predicate and running quantize_oq_streaming tensor-by-tensor (~3–4 GB host RAM for a 360 GB source).
• 0 fallbacks across 2,134 predicate decisions — the map covers the model completely; nothing silently took oMLX defaults.
• Bit placement follows weight mass vs. sensitivity: 68 GB of 4-bit experts (≈half of trunk prefill FLOPs, and the NAX gs64 tensor-op path), 8-bit everywhere small and quality-critical. Depth-coupled paths get extra precision — the GDN recurrent state (36 of 48 layers) accumulates error over sequence length, so its projections stay 8-bit, and the sparse-attention indexer, which selects the tokens the 12 full-attention layers see, stays exact bf16 (it is only ~20 MB). Per-token paths (routed experts, PLE table) tolerate 4-bit.
• Byte split as measured from the checkpoint: 91.78 GB packed quants + 14.38 GB bf16 scales/biases
  • 1.83 GB kept exact.
• The n-gram table is 4-bit gs32 because it has to be — PLE rows are 160 wide (8 heads × 20), and 64 does not divide 160. Same class of constraint as the 27 vision linear_fc2, which stay bf16 because input dim 4304 admits no supported group size.
• MTP head (2.61B params / 2.79 GB) produced by a text-only pass and grafted in as model-mtp.safetensors (58 tensors), since the stock validator rejects a qwen4_exp recipient.
• 2026-09-03: MTP expert MLP re-quantized 4-bit → 8-bit gs64 (requant_mtp_mlp8.py, +1.26 GB, round-trip error 0.007 vs ~0.10 at 4-bit), rebuilt from the bf16 source. The MTP head only drafts — the trunk verifies every token, so its precision affects speculative acceptance rate (decode speed), never output quality. Previous 4-bit shard kept as model-mtp.safetensors.bak-mlp4bit.
• Tokenizer, chat template and configs are the source's own files.
• Not re-calibrated beyond the quantizer's own pass — no training, fine-tuning, or weight editing.

Requires oMLX

Needs oMLX ≥ 0.6.3rc3 (model_type: qwen4_exp, VL engine, pre-quantization sanitize). Two things are engine-level model settings, not request parameters: mtp_enabled (Lightning MTP decode — the "enable_mtp" request field is ignored) and PLE mode.

Set PLE to mode="mmap". With the n-gram table resident the build loads in 94 s and holds 107.6 GB hot, which macOS killed under memory pressure on a 128 GB desktop. mmap loads in ~11 s and resides at ~76.9 GB (was 75.7 before the 8-bit MTP MLP), which is what has been serving stably. Since the PLE table is a raw mmap with no row cache, keep RAM pressure low so its hot pages stay in page cache (oMLX hot_cache_max_size lowered 12 → 6 GB for this reason).

Measured on M5 Max 128 GB / oMLX 0.6.4: 30.7 tok/s decode aggregate (25,505 tokens over 30 agentic tool-calling turns, 12.5k → 79k context, 0 errors), 91.7% prompt-cache hits. Trunk KV is only 24 KB/token (12 of 48 layers cache KV) — 6.4 GB at 262k, 3.2 GB at 128k. Speculative-prefill token pruning does not apply here: GDN state needs every token and pruning breaks n-gram hash adjacency.