Qwen3.6-35B-A3B 128-Expert Pruned GGUF

Experimental research artifact. This is a 50% expert-pruned model, not an official Qwen release. The pruning calibration is small and coding/agent biased. Test the Q8 artifact on your workload before trusting a lower quant.

This repository contains Q8 through one-bit-class GGUF variants of Qwen/Qwen3.6-35B-A3B. The routed MoE was reduced from 256 to 128 experts per layer while retaining the original top-k 8 active experts per token. Pruning reduces stored parameters and memory, but it does not reduce the eight active expert matmuls per token.

Quant files

GGUF has no conventional Q7_K family, so the ladder follows the supported practical families: Q8, Q6, Q5, Q4, Q3, Q2/IQ2, then IQ1. IQ1_M and IQ1_S are both included because they are distinct one-bit-class formats.

Requantization warning: every file below Q8_0 was produced from the pruned Q8_0 artifact with --allow-requantize, not from F16/BF16. The quantizer itself warns that this can reduce quality more than direct quantization from 16/32-bit weights. A remapped 128-expert importance matrix was used for every lower quant.

What was pruned

This checkpoint is not a direct REAP implementation. REAP motivated the experiment, but the Qwen selection used low router-use frequency because that criterion performed better than REAP in the preceding local Granite-MoE control experiment. See the REAP paper and official Cerebras implementation for the router-weighted activation-norm method that inspired this work.

The selection procedure was:

1. Run 512 coding/agent calibration tokens through all 40 MoE layers.
2. Record 20,480 layer-token observations and each token's top-8 routed expert IDs.
3. In each layer, prune the 128 experts with the lowest routed top-k frequency; break ties deterministically by original expert ID.
4. Copy retained expert rows and corresponding router rows in ascending original-ID order. Update expert_count from 256 to 128.
5. Preserve the raw quantized bytes of retained expert tensors. Shared experts and all other tensors remain byte-identical to the source Q8.
6. Re-open the output and fully hash-check all 733 tensors before publication.

The exact retained IDs, per-expert route counts, source trace hash, and transformation report are included under method/.

Evaluation

The first table compares the unpruned Q8 baseline, the 50%-pruned Q8, and the existing all-expert mixed-Q2 deployment reference on the same eight deterministic coding, JSON, math, tool-call, and context checks.

Exact-output equality is reported separately from task correctness. Runtime numbers are specific to a local RTX 3060 Ti + RTX 2080 SUPER + RTX 3080 setup; the unpruned Q8 baseline used CPU expert offload, while the smaller models fit across all three GPUs.

The complete per-prompt outputs and timing records are in evaluation/. These checks are a smoke/acceptance suite, not a replacement for HumanEval, BFCL, MMLU, long-context, or safety evaluation.

All-quant functional and speed benchmark

All functional-speed rows except IQ1_M used the local ik_llama CUDA build. That build aborts on IQ1_M with Unhandled type iq1_m (29), so IQ1_M was bench-tested with the local TurboQuant llama.cpp build instead. Its speed is therefore not directly comparable to the other rows; the engine and full command are preserved in evaluation/quant-ladder-qa-results.json.

Held-out perplexity by quant

Perplexity used the included evaluation/qwen36_ppl_heldout.txt, which is separate from both the pruning router trace and the importance-matrix calibration. Values are most useful as relative degradation within this table, not as a cross-model leaderboard.

Practical selection from this sweep

• Q8_0 is the reference for judging pruning independently of further quantization.
• Q5_K_M is the measured recommendation for this local coding/agent workload: 8/8 checks, +1.05% held-out PPL, and 118.28 median decode tok/s.
• Q4_K_M is the smaller viable choice: 7/8 checks, +2.38% PPL, and 125.22 tok/s.
• Q3 and below are research/space-first builds, not general recommendations; their perplexity degradation is material even when a tiny smoke suite passes.

Usage

llama-server \
  -m Qwen3.6-35B-A3B-128E-Pruned-Q4_K_M.gguf \
  --jinja --n-gpu-layers 99 --ctx-size 32768

Use a recent llama.cpp-compatible runtime with qwen35moe support. The Q8 file is the best reference for deciding whether pruning itself is acceptable; move down the quant ladder only after that comparison.

Limitations

• The 512-token selection trace is small and strongly biased toward coding/agent work.
• Experts rarely used by that trace may be important for languages, domains, knowledge, reasoning styles, or safety behavior absent from calibration.
• There was no recovery fine-tuning after pruning.
• Frequency pruning measures router use, not expert activation magnitude or REAP saliency.
• Halving stored experts does not halve decode compute because top-k remains 8.
• Low-bit requants compound pruning risk with Q8-to-lower-bit requantization error.
• Do not use this experiment for high-stakes decisions without independent evaluation.

Reproducibility files

• quant-manifest.json: source, imatrix, quantizer hashes, commands, sizes, and hashes.
• SHA256SUMS: complete model-file checksums.
• method/expert-selection.json: retained/pruned IDs and route counts for every layer.
• method/pruning-report.json: Q8 rewrite and verification report.
• method/imatrix-remap-report.json: 256-to-128 imatrix transformation report.
• evaluation/: deterministic QA outputs and timing evidence.

License and attribution

The GGUF metadata identifies the base license as Apache-2.0. This derivative keeps that license metadata. The source Q8 metadata credits Unsloth as its quantizer. Review the upstream model card and license before redistribution or use.