Qwen3.8-Flash-Next REAP-384 (MLX, 4-bit)
| Disk | Resident memory | HumanEval pass@1 | |
|---|---|---|---|
| Base Q4 (512 experts) | 98 GB | 97 GB | 93.9% |
| This build (384 experts) | 80 GB (-18%) | 51 GB | 92.1% |
The resident figure is measured with the n-gram table served from NVMe (see below); loaded fully in memory this build is ~80 GB resident.
Qwen3.8-Flash-Next with 384 of 512 experts per MoE layer, pruned with REAP saliency calibrated on the quantized weights, on the machine that serves them. This is the shallowest cut in our pruning ladder that we publish: at 1.8 points off the stock conversion on HumanEval it is the closest-to-stock option when your disk budget allows 80 GB. If you want the best quality-per-GB instead, take sh0wie/Qwen3.8-Flash-Next-REAP-288-MLX-4bit (68 GB, 91.5%). Loads on stock mlx-vlm with no patches.
- 180B-parameter class: 125B main model, 51B n-gram embedding table, 48 layers alternating Gated DeltaNet and Qwen sparse attention, each with a 384-expert MoE routing top-10
- Affine 4-bit throughout (group size 64; n-gram table group size 32)
- Multimodal weights (vision tower) are intact but only text quality has been evaluated
Run it
Requires mlx-vlm with qwen4_exp MTP support (git main after 2026-08-27, or
any release that includes it):
pip install git+https://github.com/Blaizzy/mlx-vlm.git
Generate:
python -m mlx_vlm.generate \
--model sh0wie/Qwen3.8-Flash-Next-REAP-384-MLX-4bit \
--prompt "Refactor this function to add input validation." \
--max-tokens 512
Serve (OpenAI-compatible):
python -m mlx_vlm.server \
--model sh0wie/Qwen3.8-Flash-Next-REAP-384-MLX-4bit --port 8080
Speculative decoding with the model's own MTP head, using the companion drafter sh0wie/Qwen3.8-Flash-Next-MTP-Drafter-MLX-bf16:
python -m mlx_vlm.generate \
--model sh0wie/Qwen3.8-Flash-Next-REAP-384-MLX-4bit \
--draft-model sh0wie/Qwen3.8-Flash-Next-MTP-Drafter-MLX-bf16 \
--draft-kind mtp \
--prompt "..." --max-tokens 512
A note on speculative speed: the drafter's acceptance rate is healthy (~44-68% depending on sampling), but the net speedup depends on how cheaply your hardware runs the verification pass. M5-class GPUs report 1.5-2.6x; on M4 it is roughly break-even. Quality is unaffected either way, since the target model verifies every drafted token.
Where 384 sits on the ladder
We measured an eleven-point pruning ladder; 384 is the runner-up, and the right choice when you are optimizing for quality rather than footprint:
| Build (experts) | Disk | HumanEval pass@1 |
|---|---|---|
| 512 (stock conversion) | 98 GB | 93.9% |
| 384 (this build) | 80 GB | 92.1% |
| 320 | 72 GB | 90.9% |
| 288 (efficiency pick) | 68 GB | 91.5% |
| 256 | 65 GB | 88.4% |
All legs ran the same harness on the same machine: 164 HumanEval problems,
unit-test verified, one run per build. Routing width is untouched at the
trained top-10. The kept-expert manifest ships in this repo as
reap_kept_experts.json, which makes the prune reproducible from the source
conversion.
The NVMe n-gram mode
Per token the model reads only a few hundred bytes of the 51B n-gram table, so the table does not need to be resident. A row-granular disk-read patch (ours, not yet upstream in mlx-vlm) serves it from NVMe with logits bit-identical to the in-memory path, dropping resident memory from ~80 GB to 51 GB measured (MLX peak 50.6 GiB during a short greedy run).
Provenance and what was fixed
Qwen/Qwen3.8-Flash-Next: upstream weights- Sawfwair/Qwen3.8-Flash-Next-MLX-4bit: MLX affine 4-bit conversion (group size 64; n-gram table group size 32)
- This build: REAP expert pruning 512 -> 384 per layer, calibrated on-device over ~686K tokens of agentic-coding traffic
Two defects of the source conversion are corrected in the weights, so no
loader patches are needed: RMSNorm tensors stored un-centered (+1) are
re-centered to the zero-centered convention the runtime's (1 + w) norm
expects, and the n-gram table tensors plus their per-tensor quantization
overrides are renamed shard_N -> shards.N to match the runtime module
path. Everything else is byte-identical to the pruned source. Stock-runtime
logits on this build match our patched-runtime reference (max abs diff
0.0 at the final prefill position).
Limitations
- Calibration reflects one team's agentic-coding distribution. Retention numbers should not be read as general-domain; domains far from code may degrade more.
- Single-run evaluations, no confidence intervals. Differences of a point or two between neighboring builds are within noise.
- Vision input is untested after pruning.
License
Qwen Community License 1.0, inherited from the base model; see LICENSE.