Niwaki-Qwen3.6-24B-A3B-v3.0-gguf
GGUF build of Niwaki-Qwen3.6-24B-A3B-v3.0-mlx —
Qwen3.6-35B-A3B with every routed expert stored at its own precision or
removed, its program-state tracking intact and ten major languages kept —
for llama.cpp.
Requires the neopolita-llama.cpp
fork at tag v3.0 or later (see Usage): stock llama.cpp cannot
represent per-expert precision yet.
Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts. A paper with the full method and measurements is coming soon.
Project page, with the releases and their measurements side by side: niwakiai.com
File
| file | size | wt2 ppl (llama.cpp, 512-ctx) ↓ | C4 ppl ↓ |
|---|---|---|---|
| Niwaki-Qwen3.6-24B-A3B-v3.0.gguf | 10.8 GB | 8.18 ±0.05 | 12.40 ±0.09 |
The same model as the MLX repo. Scored on identical tokens (WikiText-2,
40 windows of 512 tokens), this file reads 7.255 against the MLX
artifact's 7.257, and within 0.7% of it on FLORES-200 in each of English,
Spanish, French, German, Russian, Chinese, Japanese, Korean, Arabic and
Hindi — the precision layout carries over expert by expert (the
routed experts stored at 2-bit sit in Q2_K, those at 3 and 4 bits in Q4_K,
quantized with an importance matrix collected on text in those ten
languages; the backbone in Q8_0). The
program-state battery, run through llama-server on the suite's items,
matches the MLX artifact within two items per family: two-hop tracking
0.983 (MLX 0.975), the distractor family 0.933 (0.917), two-step arithmetic
0.600 (0.567), multi-hop recall 0.812 (0.812).
Reference Qwen3.6-35B-A3B at Q8_0 measures 6.95 on WikiText-2 under
the identical protocol (llama-perplexity, WikiText-2 test, 512-token
windows). These llama.cpp numbers are not directly comparable to the MLX
repo's 2048-window benchmarks; the canonical measurements (task suites,
gsm8k, the program-state battery) live there.
Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated, one session): 978 tok/s prompt, 63 tok/s decode; the v2.3-4bit UD-Q3K build 1,494 / 82 tok/s in the same session. Each layer runs its experts in two precision tiers, which costs speed against a single-bank file; the file still runs at about 1.4× the decode speed of the intact model's dynamic quants of this size (UD-IQ2_XXS / UD-IQ2_M: 45 tok/s, 808–832 prompt, measured in an earlier session in which the v2.3 build read the same 82 tok/s). The MLX repo decodes at 80 tok/s on the same machine.
Against the earlier Niwaki builds and the intact model
Perplexity under the protocol above (llama-perplexity, 512-token windows, WikiText-2 test and a 512-chunk C4 slice), all measured on the same machine with the same build family:
| file | size | wt2 ppl ↓ | C4 ppl ↓ |
|---|---|---|---|
| Qwen3.6-23B-A3B-Niwaki-v2.3-2bit UD-Q3K | 11.2 GB | 9.47 | 14.55 |
| Qwen3.6-23B-A3B-Niwaki-v2.3-4bit UD-Q3K | 11.2 GB | 8.39 | 13.31 |
| Qwen3.6-35B-A3B-UD-IQ2_XXS (intact, Unsloth) | 10.8 GB | 7.69 | 11.96 |
| Qwen3.6-35B-A3B-UD-IQ2_M (intact, Unsloth) | 11.5 GB | 7.59 | 11.83 |
| this repo | 10.8 GB | 8.18 | 12.40 |
At the size of v2.3's UD-Q3K builds this file reads 14–15% lower perplexity than the 2-bit one and 3% / 7% lower than the 4-bit one. Read the intact model as a split: its dynamic quants of this size read 6–7% lower WikiText-2 and 4–5% lower C4 perplexity; this file carries the MLX artifact's layout, its measured program-state behaviour and its ten languages (above) — the intact quants' were not measured here.
Model dimensions
| total / active parameters | 24.1B / at most ~3.8B (8 routed + 1 shared expert per layer; a removed expert's slot is dropped) |
| layers / routed experts / top-k | 40 / 256 / 8 (every layer keeps its full router) |
| routed experts stored | 10,240: 5,141 in Q2_K (2-bit in MLX), 1,611 in Q4_K (3- and 4-bit in MLX), 3,488 removed |
| expert intermediate size | 512 (unchanged) |
| backbone | Q8_0 (embeddings and output head Q8_0) |
| context | as base model |
| conversion note | text only; speculative-decoding (MTP) draft block not included |
Usage
git clone --branch v3.0 https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion
hf download neopolita/Niwaki-Qwen3.6-24B-A3B-v3.0-gguf Niwaki-Qwen3.6-24B-A3B-v3.0.gguf --local-dir .
./build/bin/llama-cli -m Niwaki-Qwen3.6-24B-A3B-v3.0.gguf
# or serve:
./build/bin/llama-server -m Niwaki-Qwen3.6-24B-A3B-v3.0.gguf
Why a fork
Every routed expert of this model has its own precision, and some are
removed. Stock llama.cpp stores a layer's experts in one tensor with one
quantization type and expects all of them to exist; the fork lets a
qwen35moe layer store its experts in precision tiers — one tensor per
quantization type, with a small per-tier table of which experts it holds —
under the full router, so the routing is the base model's and a removed
expert contributes nothing. Files without tiers, including every earlier
Niwaki GGUF, load and run exactly as before. Files produced by the fork's
converter load only in the fork; upstreaming is planned.
Canonical benchmarks, the progress table since the first generation, and the MLX-native artifact: Niwaki-Qwen3.6-24B-A3B-v3.0-mlx. Family: v3.0 30B / 27B · v2.3 4-bit / 2-bit · v2.2 4-bit / 2-bit · v2.1 4-bit / 2-bit · v2 4-bit / 2-bit · first-generation 27B-2bit · 19B-2bit · 11B-4bit.
Base model by the Qwen team (Apache 2.0); compression and GGUF builds by the Niwaki project, 2026-09.