Niwaki-Qwen3.6-27B-A3B-v3.0-gguf
GGUF build of Niwaki-Qwen3.6-27B-A3B-v3.0-mlx —
Qwen3.6-35B-A3B with every routed expert stored at its own precision or
removed, its program-state tracking intact and ten major languages kept —
for llama.cpp.
Requires the neopolita-llama.cpp
fork at tag v3.0 or later (see Usage): stock llama.cpp cannot
represent per-expert precision yet.
Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts. A paper with the full method and measurements is coming soon.
Project page, with the releases and their measurements side by side: niwakiai.com
File
| file | size | wt2 ppl (llama.cpp, 512-ctx) ↓ | C4 ppl ↓ |
|---|---|---|---|
| Niwaki-Qwen3.6-27B-A3B-v3.0.gguf | 12.8 GB | 7.52 ±0.05 | 11.72 ±0.09 |
The same model as the MLX repo. Scored on identical tokens (WikiText-2,
40 windows of 512 tokens), this file reads 6.648 against the MLX
artifact's 6.639 (+0.1%), and within 0.6% of it on FLORES-200 in each of
English, Spanish, French, German, Russian, Chinese, Japanese, Korean,
Arabic and Hindi — the precision layout carries over expert by expert (the
routed experts stored at 2-bit sit in Q2_K, those at 3 and 4 bits in Q4_K,
quantized with an importance matrix collected on text in those ten
languages; the backbone in Q8_0). The
program-state battery, run through llama-server on the suite's items,
matches the MLX artifact within two items per family: two-hop tracking
0.992 (MLX 0.992), the distractor family 0.950 (0.950), two-step arithmetic
0.650 (0.617), multi-hop recall 0.938 (0.938).
Reference Qwen3.6-35B-A3B at Q8_0 measures 6.95 on WikiText-2 under
the identical protocol (llama-perplexity, WikiText-2 test, 512-token
windows). These llama.cpp numbers are not directly comparable to the MLX
repo's 2048-window benchmarks; the canonical measurements (task suites,
gsm8k, the program-state battery) live there.
Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated, one session): 959 tok/s prompt, 64 tok/s decode; the v2.3-4bit UD-Q3K build 1,494 / 82 tok/s in the same session. Each layer runs its experts in two precision tiers, which costs speed against a single-bank file; the file still runs at about 1.4× the decode speed of the intact model's dynamic quant of about the same size (UD-Q2_K_XL: 45 tok/s, 756 prompt, measured in an earlier session in which the v2.3 build read the same 82 tok/s). The MLX repo decodes at 77 tok/s on the same machine.
Against the earlier Niwaki builds and the intact model
Perplexity under the protocol above (llama-perplexity, 512-token windows, WikiText-2 test and a 512-chunk C4 slice), all measured on the same machine with the same build family:
| file | size | wt2 ppl ↓ | C4 ppl ↓ |
|---|---|---|---|
| Qwen3.6-23B-A3B-Niwaki-v2.3-4bit UD-Q3K | 11.2 GB | 8.39 | 13.31 |
| Qwen3.6-23B-A3B-Niwaki-v2.3-4bit Q4_K_M | 14.0 GB | 8.31 | 13.20 |
| Qwen3.6-35B-A3B-UD-IQ2_M (intact, Unsloth) | 11.5 GB | 7.59 | 11.83 |
| Qwen3.6-35B-A3B-UD-Q2_K_XL (intact, Unsloth) | 12.3 GB | 7.48 | 11.67 |
| Qwen3.6-35B-A3B-UD-IQ3_XXS (intact, Unsloth) | 13.2 GB | 7.37 | 11.44 |
| this repo | 12.8 GB | 7.52 | 11.72 |
Between v2.3's two 4-bit builds in size, this file reads 10% lower WikiText-2 and 12% lower C4 perplexity than the UD-Q3K build. Against the intact model's dynamic quant of about the same size (UD-Q2_K_XL, 12.3 GB) it is level on perplexity (7.52 / 11.72 against 7.48 / 11.67), while keeping the MLX artifact's program-state behaviour shown above.
Model dimensions
| total / active parameters | 27.0B / at most ~3.8B (8 routed + 1 shared expert per layer; a removed expert's slot is dropped) |
| layers / routed experts / top-k | 40 / 256 / 8 (every layer keeps its full router) |
| routed experts stored | 10,240: 4,684 in Q2_K (2-bit in MLX), 2,984 in Q4_K (3- and 4-bit in MLX), 2,572 removed |
| expert intermediate size | 512 (unchanged) |
| backbone | Q8_0 (embeddings and output head Q8_0) |
| context | as base model |
| conversion note | text only; speculative-decoding (MTP) draft block not included |
Usage
git clone --branch v3.0 https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion
hf download neopolita/Niwaki-Qwen3.6-27B-A3B-v3.0-gguf Niwaki-Qwen3.6-27B-A3B-v3.0.gguf --local-dir .
./build/bin/llama-cli -m Niwaki-Qwen3.6-27B-A3B-v3.0.gguf
# or serve:
./build/bin/llama-server -m Niwaki-Qwen3.6-27B-A3B-v3.0.gguf
Why a fork
Every routed expert of this model has its own precision, and some are
removed. Stock llama.cpp stores a layer's experts in one tensor with one
quantization type and expects all of them to exist; the fork lets a
qwen35moe layer store its experts in precision tiers — one tensor per
quantization type, with a small per-tier table of which experts it holds —
under the full router, so the routing is the base model's and a removed
expert contributes nothing. Files without tiers, including every earlier
Niwaki GGUF, load and run exactly as before. Files produced by the fork's
converter load only in the fork; upstreaming is planned.
Canonical benchmarks, the progress table since the first generation, and the MLX-native artifact: Niwaki-Qwen3.6-27B-A3B-v3.0-mlx. Family: v3.0 30B / 24B · v2.3 4-bit / 2-bit · v2.2 4-bit / 2-bit · v2.1 4-bit / 2-bit · v2 4-bit / 2-bit · first-generation 27B-2bit · 19B-2bit · 11B-4bit.
Base model by the Qwen team (Apache 2.0); compression and GGUF builds by the Niwaki project, 2026-09.