Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-mlx
Qwen3.8-Flash-Next pruned to 102B total / ~5.3B active parameters, stored
at 3-bit — 37.5 GB on disk, for 48 GB Macs. The program-state behaviours the
first-generation 99B lost (variable tracking through reassignments and past
distractor lines) are back at the unpruned model's level, at the same
perplexity and 0.8 GB more. Stock mlx-vlm plus the small loader shipped in
this repo.
Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts. A paper with the full method and measurements is coming soon.
Project page, with the releases and their measurements side by side: niwakiai.com
What changed from the first generation
The earlier 99B removed the routed experts of 24 layers and narrowed every remaining expert. This generation keeps that layout and its byte budget, keeps a small set of full-width experts (26–31 per layer) on those 24 layers, and follows the pruning with recovery training. Same perplexity protocol, same task suite:
| model | size | wt2 ppl ↓ | C4 ppl ↓ | task avg ↑ | tracking / distractors / arithmetic (see below) |
|---|---|---|---|---|---|
reference Qwen3.8-Flash-Next (4-bit) | 111.5 GB | 5.06 | 10.97 | 0.759 | 0.96 / 0.96 / 0.91 |
| Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit (first generation) | 36.7 GB | 7.88 | 14.71 | 0.733 | 0.67 / 0.68 / 0.82 |
| this model | 37.5 GB | 7.84 | 14.80 | 0.738 | 0.96 / 0.96 / 0.87 |
Larger sibling for 64 GB Macs: Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-mlx (44.8 GB, wt2 6.36, task average 0.762).
Benchmarks
Evaluation protocol: WikiText-2 and C4 perplexity over 64 × 2048-token windows (fp32 loss, no chat template); task average over arc_easy, hellaswag, piqa, winogrande and boolq (zero-shot, 500 examples each), paired against the reference on the identical examples; KL divergence of the next-token distribution to the reference over 16 × 2048 windows.
| model | expert storage | wt2 ppl ↓ | C4 ppl ↓ | KLD ↓ | task avg ↑ | retention | generation d2 (avg/min) |
|---|---|---|---|---|---|---|---|
| reference (4-bit) | 75.5 GB | 5.06 | 10.97 | — | 0.759 | 100% | 0.78 / 0.62 |
| this model | 20.0 GB (0.26×) | 7.84 | 14.80 | 0.96 | 0.738 | 97.2% | 0.70 / 0.50 |
Program-state behaviours
A 964-item suite of synthetic program-state tasks (variable tracking through reassignments, tracking with distractor statements, two-step arithmetic, multi-hop lookups), scored by forced choice between candidate answers on two held-out seeds; means over the two seeds. This is where compression usually breaks first.
| model | variable tracking | tracking with distractors | two-step arithmetic | multi-hop |
|---|---|---|---|---|
| reference (4-bit) | 0.95 / 0.96 | 0.96 | 0.91 | 1.00 |
| 99B (first generation, one seed) | 0.75 / 0.58 | 0.68 | 0.82 | 1.00 |
| this model | 0.95 / 0.96 | 0.96 | 0.87 | 0.97 |
Over the 1,928 scored items, this model answers 15 items wrong that the reference gets right and 7 right that the reference gets wrong; nine of the fifteen are two-step arithmetic, the one family still below the reference.
Generation quality is measured, not assumed: an 8-prompt battery (code,
reasoning, chat, creative; 600-token sampled generations) scored by bigram
diversity. With thinking on, this model reasons at length and uses more of
the 600-token budget than the reference, which lowers its diversity score
(0.70 vs 0.78); use enable_thinking=False for short-form tasks.
Model dimensions
| total parameters | 102B (from ~177B) |
| active parameters per token | ~5.3B (from ~6.7B) |
| layers / routed experts / top-k | 48 / 512 / 10 (24 layers keep all 512 experts, narrowed; 24 layers keep 26–31 full-width experts each, and their routed path contributes only when the router picks one of them) |
| expert intermediate size | 448 on the 24 full layers (from 640); 640 for the kept experts of the other 24 |
| expert precision | 3-bit (group 64) |
| n-gram embedding table (PLE) precision | 2-bit (group 128) |
| backbone precision | 4-bit (untouched) |
| whole-model size on disk | 37.5 GB |
| runs in | 48 GB unified memory (see Usage for the GPU memory limit) |
| modality | text only (vision tower not included) |
Usage (MLX, Apple Silicon)
Requires mlx-vlm >= 0.7.0rc0 and mlx >= 0.32, plus niwaki_flash_load.py
from this repo (it tells stock mlx-vlm about the artifact's layout).
pip install -U mlx mlx-vlm transformers
hf download neopolita/Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-mlx --local-dir ./niwaki-flash-102b
import mlx_vlm
from niwaki_flash_load import load # niwaki_flash_load.py from this repo
model, processor = load("./niwaki-flash-102b")
messages = [{"role": "user", "content": "Explain hyper-connections in two sentences."}]
text = processor.tokenizer.apply_chat_template(messages, add_generation_prompt=True,
tokenize=False, enable_thinking=False)
print(mlx_vlm.generate(model, processor, text, max_tokens=300, verbose=False).text)
enable_thinking=False skips the reasoning preamble; leave it on for harder
prompts. On an M4 Max: ~31 tokens/s, ~38 GB peak.
On a 48 GB Mac that peak leaves little room under macOS's default GPU memory
limit (about three quarters of RAM). If generation slows down or fails at long
context, raise the limit until the next reboot with
sudo sysctl iogpu.wired_limit_mb=42000.
GGUF builds for llama.cpp (CUDA, Vulkan, Metal, CPU; they need the neopolita-llama.cpp fork): Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-gguf (UD-Q3K 43.9 GB recommended, UD-Q4K 55.9 GB).
Notes and limitations
- Compression trades quality: this model sits at ~97% of the reference task average and 1.55× its perplexity, with the program-state behaviours above at or near the reference's level. Fluent and on-topic; numbers and niche technical facts are less reliable than the reference.
- Two-step arithmetic is the family that remains below the reference (0.87 against 0.91); the larger 119B sibling holds it (0.93).
- Evaluated text-only on English-web-heavy data; the base model's biases are inherited and rare-domain behaviour of pruned experts is less tested.
- First generation: 99B · 113B. The Niwaki family on Qwen3.6-35B-A3B: v2.3 4-bit / 2-bit.
Base model by the Qwen team (see license); 4-bit MLX conversion by mlx-community; pruning and recovery training by the Niwaki project, 2026-09. Method details: paper coming soon.