Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-gguf
GGUF builds of Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-mlx — Qwen3.8-Flash-Next pruned to 99B total / ~5.2B active parameters — for llama.cpp on CUDA, Vulkan, Metal and CPU. Requires the neopolita-llama.cpp fork (see Usage): stock llama.cpp cannot represent this model's layout yet.
Niwaki (庭木): routed experts pruned using statistics of their routed tokens over a mixed web/code/chat/reasoning corpus, reconstructed to compensate, then briefly distilled from the full model, and stored at low precision. A paper with the full method is coming soon.
Files
| file | size | wt2 ppl (llama.cpp, 512-ctx) |
|---|---|---|
| Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-UD-Q3K.gguf (recommended) | 42.5 GB | 8.79 ±0.06 |
| Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-UD-Q4K.gguf | 54.1 GB | 8.61 ±0.06 |
Protocol: llama-perplexity, WikiText-2 test, 512-token windows. These llama.cpp numbers are not directly comparable to the MLX repo's 2048-window benchmarks; the canonical measurements (against the reference model) live there.
Both builds are quantized structure-aware, mirroring the MLX artifact's native allocation: the always-active backbone (attention, linear attention, hyper-connections, shared experts) at Q6_K, embeddings and output at Q8_0, the pruned routed experts on a compact carrier (Q3_K, or Q4_K in the UD-Q4K file), the n-gram embedding table at Q2_K (Q3_K in UD-Q4K), and the distilled output maps at Q8_0.
Model dimensions
| total / active parameters | 99B / ~5.2B |
| layers / routed experts / top-k | 48 / 512 / 10 (routed experts on 24 of the 48 layers) |
| expert intermediate size | 512 (448 used; zero-padded for the quantization block size) |
| context | as base model |
| modality | text only (vision tower not included; MTP draft head not included) |
Usage
git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion
hf download neopolita/Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-gguf Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-UD-Q3K.gguf
On an M4 Max (Metal) the UD-Q3K file decodes at ~30 tokens/s.
Why a fork
The fork adds three small, general features to llama.cpp's qwen4exp
architecture, needed to store this model at its real size:
- routed banks may be absent on a layer (the router and shared expert stay); stock llama.cpp needs uniform expert tensors, which would add ~17 GB of all-zero experts here;
- a packed layout for the per-layer n-gram embedding table (
ple.row_pack), so block-256 quantization types apply to its 160-wide rows; otherwise the table alone is 28.8 GB at 4.5 bits; - an optional per-layer output map (
ffn_out_map), so the distilled healing maps ship as tensors instead of being folded into 3-bit weights.
Files produced by the fork's converter load only in the fork; upstreaming is planned.
Notes and limitations
- Quality follows the MLX artifact: ~97% of the reference's task average and 1.56× its perplexity under the MLX protocol. Fluent and on-topic; numbers and niche technical facts are less reliable than the reference.
- Evaluated text-only on English-web-heavy data; the base model's biases are inherited.
- Canonical benchmarks, dimensions and the MLX-native artifact: Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-mlx.
Base model by the Qwen team (see license); pruning, distillation and GGUF builds by the Niwaki project, 2026-09.