Niwaki-Qwen3.6-30B-A3B-v3.0-gguf
GGUF build of Niwaki-Qwen3.6-30B-A3B-v3.0-mlx —
Qwen3.6-35B-A3B with every routed expert stored at its own precision or
removed, its program-state tracking intact and ten major languages kept —
for llama.cpp.
Requires the neopolita-llama.cpp
fork at tag v3.0 or later (see Usage): stock llama.cpp cannot
represent per-expert precision yet.
Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts. A paper with the full method and measurements is coming soon.
Project page, with the releases and their measurements side by side: niwakiai.com
File
| file | size | wt2 ppl (llama.cpp, 512-ctx) ↓ | C4 ppl ↓ |
|---|---|---|---|
| Niwaki-Qwen3.6-30B-A3B-v3.0.gguf | 14.1 GB | 7.25 ±0.05 | 11.63 ±0.09 |
The same model as the MLX repo. Scored on identical tokens (WikiText-2,
40 windows of 512 tokens), this file reads 6.410 against the MLX
artifact's 6.383 (+0.4%), and within 0.5% of it on FLORES-200 in each of
English, Spanish, French, German, Russian, Chinese, Japanese, Korean,
Arabic and Hindi — the precision layout carries over expert by expert (the
routed experts stored at 2-bit sit in Q2_K, those at 3 and 4 bits in Q4_K,
quantized with an importance matrix collected on text in those ten
languages; the backbone in Q8_0). The program-state battery, run through
llama-server on the suite's items, matches the MLX artifact within two
items per family: two-hop tracking 1.000 (MLX 0.992), the distractor family
0.983 (0.983), two-step arithmetic 0.550 (0.533), multi-hop recall 0.938
(0.938).
Reference Qwen3.6-35B-A3B at Q8_0 measures 6.95 on WikiText-2 under
the identical protocol (llama-perplexity, WikiText-2 test, 512-token
windows). These llama.cpp numbers are not directly comparable to the MLX
repo's 2048-window benchmarks; the canonical measurements (task suites,
gsm8k, the program-state battery) live there.
Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated, one session): 936 tok/s prompt, 64 tok/s decode; the v2.3-4bit UD-Q3K build 1,494 / 82 tok/s in the same session. Each layer runs its experts in two precision tiers, which costs speed against a single-bank file; the file still runs at about 1.5× the decode speed of the intact model's dynamic quants of similar size (UD-IQ3_XXS / UD-IQ3_S: 41–43 tok/s, 785–796 prompt, measured in an earlier session in which the v2.3 build read the same 82 tok/s). The MLX repo decodes at 82 tok/s on the same machine.
Against the earlier Niwaki builds and the intact model
Perplexity under the protocol above (llama-perplexity, 512-token windows, WikiText-2 test and a 512-chunk C4 slice), all measured on the same machine with the same build family:
| file | size | wt2 ppl ↓ | C4 ppl ↓ |
|---|---|---|---|
| Qwen3.6-23B-A3B-Niwaki-v2.3-4bit UD-Q3K | 11.2 GB | 8.39 | 13.31 |
| Qwen3.6-23B-A3B-Niwaki-v2.3-4bit Q4_K_M | 14.0 GB | 8.31 | 13.20 |
| Qwen3.6-35B-A3B-UD-Q2_K_XL (intact, Unsloth) | 12.3 GB | 7.48 | 11.67 |
| Qwen3.6-35B-A3B-UD-IQ3_XXS (intact, Unsloth) | 13.2 GB | 7.37 | 11.44 |
| Qwen3.6-35B-A3B-UD-IQ3_S (intact, Unsloth) | 13.7 GB | 7.39 | 11.43 |
| this repo | 14.1 GB | 7.25 | 11.63 |
At the size of v2.3's Q4_K_M build this file reads 13% lower WikiText-2 and 12% lower C4 perplexity. Against the intact model's dynamic quants of similar size it is 2% lower on WikiText-2 and 2% higher on C4 — even on perplexity — while keeping the MLX artifact's program-state behaviour shown above.
Model dimensions
| total / active parameters | 30.5B / at most ~3.8B (8 routed + 1 shared expert per layer; a removed expert's slot is dropped) |
| layers / routed experts / top-k | 40 / 256 / 8 (every layer keeps its full router) |
| routed experts stored | 10,240: 5,628 in Q2_K (2-bit in MLX), 3,145 in Q4_K (3- and 4-bit in MLX), 1,467 removed |
| expert intermediate size | 512 (unchanged) |
| backbone | Q8_0 (embeddings and output head Q8_0) |
| context | as base model |
| conversion note | text only; speculative-decoding (MTP) draft block not included |
Usage
git clone --branch v3.0 https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion
hf download neopolita/Niwaki-Qwen3.6-30B-A3B-v3.0-gguf Niwaki-Qwen3.6-30B-A3B-v3.0.gguf --local-dir .
./build/bin/llama-cli -m Niwaki-Qwen3.6-30B-A3B-v3.0.gguf
# or serve:
./build/bin/llama-server -m Niwaki-Qwen3.6-30B-A3B-v3.0.gguf
Why a fork
Every routed expert of this model has its own precision, and some are
removed. Stock llama.cpp stores a layer's experts in one tensor with one
quantization type and expects all of them to exist; the fork lets a
qwen35moe layer store its experts in precision tiers — one tensor per
quantization type, with a small per-tier table of which experts it holds —
under the full router, so the routing is the base model's and a removed
expert contributes nothing. Files without tiers, including every earlier
Niwaki GGUF, load and run exactly as before. Files produced by the fork's
converter load only in the fork; upstreaming is planned.
Canonical benchmarks, the progress table since the first generation, and the MLX-native artifact: Niwaki-Qwen3.6-30B-A3B-v3.0-mlx. Family: v3.0 27B / 24B · v2.3 4-bit / 2-bit · v2.2 4-bit / 2-bit · v2.1 4-bit / 2-bit · v2 4-bit / 2-bit · first-generation 27B-2bit · 19B-2bit · 11B-4bit.
Base model by the Qwen team (Apache 2.0); compression and GGUF builds by the Niwaki project, 2026-09.