Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-gguf
GGUF builds of Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-mlx — Qwen3.8-Flash-Next pruned to 119B total / ~5.6B active parameters, with the program-state behaviours we test (variable tracking, two-step arithmetic, multi-hop lookups) at the unpruned model's level — for llama.cpp on CUDA, Vulkan, Metal and CPU. Requires the neopolita-llama.cpp fork (see Usage): stock llama.cpp cannot represent this model's layout yet.
Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts. A paper with the full method and measurements is coming soon.
Project page, with the releases and their measurements side by side: niwakiai.com
Files
| file | size | wt2 ppl (llama.cpp, 512-ctx) |
|---|---|---|
| Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf (recommended) | 51.0 GB | 7.04 ±0.05 |
| Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-UD-Q4K.gguf | 64.3 GB | 6.88 ±0.04 |
The first generation's GGUF builds under the identical protocol:
| model (UD-Q3K builds) | size | wt2 ppl |
|---|---|---|
| Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit | 49.1 GB | 7.90 ±0.05 |
| Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit | 42.5 GB | 8.79 ±0.06 |
This build reads 11% lower perplexity than the first generation's 113B build for 1.9 GB more (the UD-Q4K file: 6.88 against 7.70).
Protocol: llama-perplexity, WikiText-2 test, 512-token windows. These llama.cpp numbers are not directly comparable to the MLX repo's 2048-window benchmarks; the canonical measurements (against the reference model: task suite, KL divergence, program-state behaviours, generation battery) live there.
Program-state check through llama.cpp (UD-Q3K file, the suite's 300
variable-tracking items, greedy decoding through llama-server): tracking
1.00 / 0.96, tracking with distractor statements 0.93 — the MLX
artifact scores 0.98 / 0.95 / 0.93 on the same items.
Both builds are quantized structure-aware, mirroring the MLX artifact's native allocation: the always-active backbone (attention, linear attention, hyper-connections, shared experts) at Q6_K (Q8_0 where a row is narrower than a 256-block), embeddings and output at Q8_0, the routed experts on a compact carrier — gate/up projections Q3_K and down projections IQ4_NL (Q4_K and Q5_0 in the UD-Q4K file; the 640-wide down rows take a 32-block type) — and the n-gram embedding table at Q2_K (Q3_K in UD-Q4K).
Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated): UD-Q3K 557 tok/s prompt, 30 tok/s decode; UD-Q4K 561 / 31 tok/s.
Model dimensions
| total / active parameters | 119B / ~5.6B |
| layers / routed experts / top-k | 48 / 512 / 10 (24 layers keep all 512 experts; 24 layers keep 16 each, and their routed path contributes only when the router picks one of them) |
| expert intermediate size | 640 (unchanged) |
| context | as base model |
| modality | text only (vision tower not included; MTP draft head not included) |
Usage
git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion
hf download neopolita/Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-gguf Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf
Support for this file layout landed in the fork together with this release:
if you built the fork for the first-generation files, git pull and rebuild.
Why a fork
These files use two small, general additions to llama.cpp's qwen4exp
architecture, needed to store this model at its real size:
- a layer may store a few of its routed experts under the full router
(
ffn_exp_ids): 24 layers keep 16 of their 512 experts. The router still scores all 512 and the top-10 is taken exactly as in the base model; an expert that is not stored contributes nothing. Stock llama.cpp needs uniform expert tensors, which would add ~28 GB of all-zero experts here; - a packed layout for the per-layer n-gram embedding table (
ple.row_pack), so block-256 quantization types apply to its 160-wide rows; otherwise the table alone is 28.8 GB at 4.5 bits.
Files produced by the fork's converter load only in the fork; upstreaming is planned.
Notes and limitations
- Quality follows the MLX artifact: the reference's task average (100.5%) and program-state behaviours at 1.26× its perplexity under the MLX protocol. Fluent and on-topic; numbers and niche technical facts are less reliable than the reference.
- With thinking on, the model reasons at length; use a no-thinking chat template setting for short-form tasks.
- Evaluated text-only on English-web-heavy data; the base model's biases are inherited.
- Canonical benchmarks, dimensions and the MLX-native artifact: Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-mlx. First generation: 113B · 99B.
Base model by the Qwen team (see license); pruning, recovery training and GGUF builds by the Niwaki project, 2026-09.