Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-mlx
Qwen3.8-Flash-Next pruned to 119B total / ~5.6B active parameters, stored
at 3-bit — 44.8 GB on disk, 10 GB below the unpruned model's best 2-bit cast
at the same perplexity and task average — for 64 GB Macs. The program-state
behaviours we test (variable tracking, two-step arithmetic, multi-hop
lookups) are at the unpruned model's level. Stock mlx-vlm plus the small
loader shipped in this repo.
Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts. A paper with the full method and measurements is coming soon.
Project page, with the releases and their measurements side by side: niwakiai.com
What changed from the first generation
The earlier 113B removed 16 routed-expert banks and narrowed every remaining expert. This generation keeps every expert at full width, removes the routed experts of 24 layers except a small set of 16 per layer that stays, and follows the pruning with recovery training. Same perplexity protocol, same task suite:
| model | size | wt2 ppl ↓ | C4 ppl ↓ | task avg ↑ | tracking / arithmetic (see below) |
|---|---|---|---|---|---|
reference Qwen3.8-Flash-Next (4-bit) | 111.5 GB | 5.06 | 10.97 | 0.759 | 0.96 / 0.91 |
| Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit (first generation) | 42.2 GB | 7.08 | 13.70 | 0.757 | 0.92 / 0.90 |
| this model | 44.8 GB | 6.36 | 12.99 | 0.762 | 0.96 / 0.93 |
Benchmarks
Evaluation protocol: WikiText-2 and C4 perplexity over 64 × 2048-token windows (fp32 loss, no chat template); task average over arc_easy, hellaswag, piqa, winogrande and boolq (zero-shot, 500 examples each), paired against the reference on the identical examples; KL divergence of the next-token distribution to the reference over 16 × 2048 windows.
| model | expert storage | wt2 ppl ↓ | C4 ppl ↓ | KLD ↓ | task avg ↑ | retention | generation d2 (avg/min) |
|---|---|---|---|---|---|---|---|
| reference (4-bit) | 75.5 GB | 5.06 | 10.97 | — | 0.759 | 100% | 0.78 / 0.62 |
| this model | 27.3 GB (0.36×) | 6.36 | 12.99 | 0.75 | 0.762 | 100.5% | 0.68 / 0.48 |
Program-state behaviours
A 964-item suite of synthetic program-state tasks (variable tracking through reassignments, tracking with distractor statements, two-step arithmetic, multi-hop lookups), scored by forced choice between candidate answers on two held-out seeds; means over the two seeds. This is where compression usually breaks first, and where the first generation's smaller sibling lost most.
| model | variable tracking | tracking with distractors | two-step arithmetic | multi-hop |
|---|---|---|---|---|
| reference (4-bit) | 0.95 / 0.96 | 0.96 | 0.91 | 1.00 |
| 113B (first generation, one seed) | 0.92 / 0.90 | 0.92 | 0.90 | 1.00 |
| this model | 0.96 / 0.97 | 0.96 | 0.93 | 0.97 |
Over the 1,928 scored items, this model answers 4 items wrong that the reference gets right and 7 right that the reference gets wrong — a paired difference indistinguishable from zero.
Against quantizing the unpruned model
The alternative to pruning is to quantize the unpruned model harder. Its best 2-bit cast under our protocol (routed experts GPTQ 2-bit, n-gram table 2-bit): 55.3 GB, wt2 6.35, task avg 0.761 — the same perplexity and task average as this model at 10.5 GB more. Its round-to-nearest 2-bit casts read 6.44 (routed experts only, 4-bit table) and lower.
Generation quality is measured, not assumed: an 8-prompt battery (code,
reasoning, chat, creative; 600-token sampled generations) scored by bigram
diversity. With thinking on, this model, like the first generation, reasons
at length and uses more of the 600-token budget than the reference, which
lowers its diversity score (0.68 vs 0.78); use enable_thinking=False for
short-form tasks.
Model dimensions
| total parameters | 119B (from ~177B) |
| active parameters per token | ~5.6B (from ~6.7B) |
| layers / routed experts / top-k | 48 / 512 / 10 (24 layers keep all 512 experts; 24 layers keep 16 each, and their routed path contributes only when the router picks one of them) |
| expert intermediate size | 640 (unchanged) |
| expert precision | 3-bit (group 64) |
| n-gram embedding table (PLE) precision | 2-bit (group 128) |
| backbone precision | 4-bit (untouched) |
| whole-model size on disk | 44.8 GB |
| runs comfortably in | 64 GB unified memory |
| modality | text only (vision tower not included) |
Usage (MLX, Apple Silicon)
Requires mlx-vlm >= 0.7.0rc0 and mlx >= 0.32, plus niwaki_flash_load.py
from this repo (it tells stock mlx-vlm about the artifact's layout).
pip install -U mlx mlx-vlm transformers
hf download neopolita/Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-mlx --local-dir ./niwaki-flash-119b
import mlx_vlm
from niwaki_flash_load import load # niwaki_flash_load.py from this repo
model, processor = load("./niwaki-flash-119b")
messages = [{"role": "user", "content": "Explain hyper-connections in two sentences."}]
text = processor.tokenizer.apply_chat_template(messages, add_generation_prompt=True,
tokenize=False, enable_thinking=False)
print(mlx_vlm.generate(model, processor, text, max_tokens=300, verbose=False).text)
enable_thinking=False skips the reasoning preamble; leave it on for harder
prompts. On an M4 Max: ~26 tokens/s, ~45 GB peak.
GGUF builds for llama.cpp (CUDA, Vulkan, Metal, CPU; they need the neopolita-llama.cpp fork): Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-gguf (UD-Q3K 51.0 GB recommended, UD-Q4K 64.3 GB).
Notes and limitations
- Compression trades quality: this model keeps the reference's task average and program-state behaviours at 1.26× its perplexity. Fluent and on-topic; numbers and niche technical facts are less reliable than the reference.
- Evaluated text-only on English-web-heavy data; the base model's biases are inherited and rare-domain behaviour of pruned experts is less tested.
- First generation: 113B · 99B. The Niwaki family on Qwen3.6-35B-A3B: v2.3 4-bit / 2-bit.
Base model by the Qwen team (see license); 4-bit MLX conversion by mlx-community; pruning and recovery training by the Niwaki project, 2026-09. Method details: paper coming soon.