MoE Expert Grading
Qwen3.6-35B-A3B · every expert gets the bit width it earns
English · 简体中文
[!WARNING] This is experimental. The only thing I measured is perplexity — one corpus, one machine, no task benchmarks. I also never sat down and compared outputs against the unquantized model line by line. So "the numbers look fine" is all I can honestly claim; it is not a promise that this is good. Only tested on Qwen3.6-35B-A3B, and speculative decoding is untested.
[!IMPORTANT] You need a patched llama.cpp. These files carry per-tier expert tensors (
*_hot/*_warm/*_cold) andmoe_expert_groupsmetadata that upstream has no idea about. A stock build gives youfailed to load model. Fork is here, with a prebuilt Windows binary on its release page.
💡 The idea
A MoE model doesn't activate all of its experts equally often. Some get called constantly, some barely at all — yet a normal quantization gives every expert the same bit width anyway. So the routed experts get quantized by how often they actually fire in real tasks: high frequency keeps more bits, low frequency gets squeezed.
📦 Pick one
| 🟢 Balanced | 🔵 Compact | |
|---|---|---|
| File | Qwen3.6-35B-A3B-MoEGraded-Balanced.gguf | Qwen3.6-35B-A3B-MoEGraded-Compact.gguf |
| Size | 17.26 GiB | 15.59 GiB |
| PPL ↓ | 1.5030 | 1.5096 |
| Hot tier | Q4_K gate/up, Q5_K down | Q4_K everywhere |
| Warm tier | Q3_K + Q4_K down | Q3_K everywhere |
| Cold tier | Q3_K | Q2_K gate/up, Q3_K down |
| Grab it when | you have the 17 GB | you don't |
Noise on this corpus runs ±0.029, so honestly these two measure the same. The difference is what the middle and cold tiers get to keep.
📊 The numbers
14 chunks × 512 tokens, -t 20, same machine throughout:
| model | size | PPL ↓ | vs F16 |
|---|---|---|---|
| F16 reference | 67 GB | 1.5053 | — |
| 🟢 Balanced | 17.26 GiB | 1.5030 | −0.002 |
| 🔵 Compact | 15.59 GiB | 1.5096 | +0.004 |
| APEX I-Compact (upstream) | 16.10 GiB | 1.5125 | +0.007 |
| Uniform Q3_K, no grading | 15.82 GiB | 1.5180 | +0.013 |
| Upstream Q4_K_S | 20 GB | 1.5167 | +0.011 |
| Upstream Q3_K_S | 15 GB | 1.5404 | +0.035 |
Balanced dipping under F16 isn't a real win — 0.002 is well inside the noise. The part worth looking at is the lower half: both files beat APEX I-Compact, and Compact does it from a file 0.51 GiB smaller.
⚡ Speed (G1 update, 2026-10)
The G1 update fuses the three per-tier mul_mat_id calls into one op: each of
the 8 fired experts is evaluated exactly once instead of three times. Measured
on an RX 5700 XT, -t 20, -p 512 -n 128:
| configuration | prompt (t/s) | generation (t/s) |
|---|---|---|
all layers on GPU (-ngl 99) | 118.7 | 11.1 |
--n-cpu-moe 36 | 217.8 | 10.2 |
CPU only (-ngl 0) | 143.0 | 6.6 |
GPU + --spec-type draft-mtp | — | 14.6 (~21 with tuned draft settings) |
Perplexity is unchanged by G1 (CPU and GPU paths agree bit-for-bit on the same corpus). For reference, the same-size standard-quant baseline (APEX I-Compact Q4_K_M) measures pp 76.5–113 and tg 12.75 on identical hardware — the remaining gap on generation comes from upstream RDNA1 Vulkan kernels, and MTP closes it.
Recommended: enable MTP
MTP uses the model's own nextn block — no sidecar file, no network access, and it is verified with tiered expert tensors:
llama-server -m Qwen3.6-35B-A3B-MoEGraded-Compact.gguf --spec-type draft-mtp -ngl 99
Draft acceptance is ~70 % (≈ 3.1 accepted tokens per round). Note the type name
is draft-mtp (plain mtp is rejected).
🔧 How it works
1 · Measure the frequency. Run an importance matrix over 5,120 calibration tokens to count how often each routed expert gets activated.
2 · Sort into three groups. Top 20% hot (51 experts), next 30% warm (77), the rest cold (128).
3 · Quantize each separately. Every group gets its own tensors and its own type, gate/up/down independently. Router, norms, attention, embeddings and shared experts stay high precision either way.
4 · Reassemble at load time. The patched loader sends every mul_mat_id
through a tier lookup table and a mask — one pass covers all three groups, and
the router still sees all 256 experts.
None of this would work if MoE were dense. Because only 8 experts fire per token, a cold expert that's slightly off barely gets touched — and that slack is what pays for the hot tier's extra bits.
▶️ Run it
Prebuilt — Windows x64, CPU + Vulkan, tested on an RX 5700 XT:
⬇ llama-fork-vulkan-graded-g1-win-x64.zip — with the G1 update
Or build it from source:
git clone --depth 1 --branch 0c1e570 https://github.com/ggml-org/llama.cpp
cd llama.cpp
patch -p1 < graded-moe-0c1e570.patch
patch -p1 < graded-moe-g1.patch
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
build/bin/llama-cli -m Qwen3.6-35B-A3B-MoEGraded-Balanced.gguf \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
-p "Write a haiku about quantization."
Sampling follows the official Qwen3.6 recommendation. Don't use greedy decoding — it hits Qwen3.6's repetition loop.
The patch, quant_recipe.py and the tiering scripts are in the
GitHub repo. README-TECH.md
there has the details and the pitfalls.
python3 quant_recipe.py --list
python3 quant_recipe.py --recipe v8 --target-mib 17673 --run --ppl
--target-mib makes the script assert the predicted size against what actually
lands on disk, so a recipe change can't silently slip through.
Built with an AI coding agent from a written design. The measurements are real; nobody outside the project has reviewed the code. Weights Apache-2.0, following Qwen3.6-35B-A3B · fork and scripts MIT.