Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-mlx
Qwen3.8-Flash-Next pruned to 113B total / ~5.4B active parameters, stored
at 3-bit — 3.1× less expert memory than the 4-bit reference, 42.2 GB on disk
(13 GB below any 2-bit cast of the unpruned model), for 64 GB Macs; keeps 99.8%
of the reference's task average. Stock mlx-vlm
plus the small loader shipped in this repo.
Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts: routed experts are pruned using statistics of their routed tokens over a mixed web/code/chat/reasoning corpus, reconstructed to compensate, then distilled from the full model on the same mixed corpus, and stored at low precision. The result is the same architecture with fewer, narrower experts.
A paper with the full method and measurements is coming soon.
Benchmarks
Evaluation protocol: WikiText-2 perplexity over 64 × 2048-token windows (fp32 loss, no chat template); task average over arc_easy, hellaswag, piqa, winogrande and boolq (zero-shot, 500 examples each), paired against the reference on the identical examples.
| model | expert storage | wt2 ppl ↓ | task avg ↑ | retention | generation d2 (avg/min) |
|---|---|---|---|---|---|
| reference (4-bit) | 75.5 GB | 5.06 | 0.759 | 100% | 0.78 / 0.62 |
| this model | 24.7 GB (0.327×) | 7.08 | 0.757 | 99.8% | 0.66 / 0.42 |
Generation quality is measured, not assumed: an 8-prompt battery (code,
reasoning, chat, creative; 600-token sampled generations) scored by bigram
diversity. Known limitation (measured): with thinking on, this model reasons at length — it used the whole 600-token budget on seven of the eight prompts where the reference stopped early on four — which lowers its diversity score (0.66 vs 0.78; minimum 0.42 on the haiku prompt); the answers themselves are on target. Use enable_thinking=False for short-form tasks.
Model dimensions
| total parameters | 113B (from ~177B) |
| active parameters per token | ~5.4B (from ~6.7B) |
| layers / routed experts / top-k | 48 / 512 / 10 (routed experts kept on 32 of the 48 layers) |
| expert intermediate size | 448 (from 640) |
| expert precision | 3-bit (group 64) |
| n-gram embedding table (PLE) precision | 2-bit (group 128) |
| backbone precision | 4-bit (untouched) |
| whole-model size on disk | 42.2 GB |
| runs comfortably in | 64 GB unified memory |
| modality | text only (vision tower not included) |
Usage (MLX, Apple Silicon)
Requires mlx-vlm >= 0.7.0rc0 and mlx >= 0.32, plus niwaki_flash_load.py
from this repo (it tells stock mlx-vlm about the artifact's layout).
pip install -U mlx mlx-vlm transformers
hf download neopolita/Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-mlx --local-dir ./niwaki-flash-113b
import mlx_vlm
from niwaki_flash_load import load # niwaki_flash_load.py from this repo
model, processor = load("./niwaki-flash-113b")
messages = [{"role": "user", "content": "Explain hyper-connections in two sentences."}]
text = processor.tokenizer.apply_chat_template(messages, add_generation_prompt=True,
tokenize=False, enable_thinking=False)
print(mlx_vlm.generate(model, processor, text, max_tokens=300, verbose=False).text)
enable_thinking=False skips the reasoning preamble; leave it on for harder
prompts. On an M4 Max: ~31 tokens/s, ~42 GB peak.
GGUF builds for llama.cpp (CUDA, Vulkan, Metal, CPU; they need the neopolita-llama.cpp fork): Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-gguf (UD-Q3K 49.1 GB recommended, UD-Q4K 62.8 GB).
Notes and limitations
- Compression trades quality: this model keeps ~99.8% of the reference task average at 1.40× its perplexity. Fluent and on-topic; numbers and niche technical facts are less reliable than the reference.
- Evaluated text-only on English-web-heavy data; the base model's biases are inherited and rare-domain behaviour of pruned experts is less tested.
- Smaller sibling for 48 GB machines: Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-mlx. The Niwaki family on Qwen3.6-35B-A3B: 27B-A3B-Niwaki-2bit · 19B-A3B-Niwaki-2bit · 11B-A3B-Niwaki-4bit.
Base model by the Qwen team (see license); 4-bit MLX conversion by mlx-community; pruning and distillation by the Niwaki project, 2026-09. Method details: paper coming soon.