Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-gguf
GGUF builds of Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-mlx — Qwen3.8-Flash-Next pruned to 102B total / ~5.3B active parameters, with the variable-tracking behaviours the first-generation 99B lost back at the unpruned model's level — for llama.cpp on CUDA, Vulkan, Metal and CPU. Requires the neopolita-llama.cpp fork (see Usage): stock llama.cpp cannot represent this model's layout yet.
Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts. A paper with the full method and measurements is coming soon.
Project page, with the releases and their measurements side by side: niwakiai.com
Files
| file | size | wt2 ppl (llama.cpp, 512-ctx) |
|---|---|---|
| Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf (recommended) | 43.9 GB | 8.75 ±0.06 |
| Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-UD-Q4K.gguf | 55.9 GB | 8.57 ±0.06 |
The first generation's GGUF builds under the identical protocol:
| model (UD-Q3K builds) | size | wt2 ppl |
|---|---|---|
| Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit | 42.5 GB | 8.79 ±0.06 |
| Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit | 49.1 GB | 7.90 ±0.05 |
Perplexity is level with the first-generation 99B build; what this generation adds is function — see the program-state table on the MLX card, where the tracking families go from 0.58–0.75 to 0.95–0.96. The larger sibling, 119B-A5B v2.4, reads 7.04 at 51.0 GB.
Protocol: llama-perplexity, WikiText-2 test, 512-token windows. These llama.cpp numbers are not directly comparable to the MLX repo's 2048-window benchmarks; the canonical measurements (against the reference model: task suite, KL divergence, program-state behaviours, generation battery) live there.
Program-state check through llama.cpp (UD-Q3K file, the suite's 300
variable-tracking items, greedy decoding through llama-server): tracking
0.96 / 0.93, tracking with distractor statements 0.92 — the MLX
artifact scores 0.98 / 0.93 / 0.90 on the same items.
Both builds are quantized structure-aware, mirroring the MLX artifact's native allocation: the always-active backbone (attention, linear attention, hyper-connections, shared experts) at Q6_K (Q8_0 where a row is narrower than a 256-block), embeddings and output at Q8_0, the routed experts on a compact carrier — Q3_K (Q4_K in the UD-Q4K file), with the 640-wide down projections of the 24 layers' kept experts on a 32-block type (IQ4_NL, or Q5_0 in UD-Q4K) — and the n-gram embedding table at Q2_K (Q3_K in UD-Q4K).
Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated): UD-Q3K 431 tok/s prompt, 26 tok/s decode; UD-Q4K 430 / 27 tok/s.
Model dimensions
| total / active parameters | 102B / ~5.3B |
| layers / routed experts / top-k | 48 / 512 / 10 (24 layers keep all 512 experts, narrowed; 24 layers keep 26–31 full-width experts each, and their routed path contributes only when the router picks one of them) |
| expert intermediate size | 512 on the 24 full layers (448 used; zero-padded for the quantization block size); 640 for the kept experts of the other 24 |
| context | as base model |
| modality | text only (vision tower not included; MTP draft head not included) |
Usage
git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion
hf download neopolita/Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-gguf Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf
Support for this file layout landed in the fork together with this release:
if you built the fork earlier, git pull and rebuild.
Why a fork
These files use three small, general additions to llama.cpp's qwen4exp
architecture, needed to store this model at its real size:
- a layer may store a few of its routed experts under the full router
(
ffn_exp_ids): 24 layers keep 26–31 of their 512 experts. The router still scores all 512 and the top-10 is taken exactly as in the base model; an expert that is not stored contributes nothing. Stock llama.cpp needs uniform expert tensors, which would add ~27 GB of all-zero experts here; - each layer's expert width comes from its tensors: the kept experts are 640 wide, every other layer's experts 512;
- a packed layout for the per-layer n-gram embedding table (
ple.row_pack), so block-256 quantization types apply to its 160-wide rows; otherwise the table alone is 28.8 GB at 4.5 bits.
Files produced by the fork's converter load only in the fork; upstreaming is planned.
Notes and limitations
- Quality follows the MLX artifact: ~97% of the reference's task average at 1.55× its perplexity under the MLX protocol, with variable tracking at the reference's level and two-step arithmetic just below it (0.87 against 0.91). Fluent and on-topic; numbers and niche technical facts are less reliable than the reference.
- With thinking on, the model reasons at length; use a no-thinking chat template setting for short-form tasks.
- Evaluated text-only on English-web-heavy data; the base model's biases are inherited.
- Canonical benchmarks, dimensions and the MLX-native artifact: Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-mlx. First generation: 99B · 113B.
Base model by the Qwen team (see license); pruning, recovery training and GGUF builds by the Niwaki project, 2026-09.