Qwen3.6-29B-REAP-Opus-Reasoning-Distill-GGUF
GGUF quantizations of a REAP-pruned (205/256 experts) Qwen3.6-35B-A3B MoE, merged with an Opus-reasoning-distilled LoRA adapter. This is the plain merge — no Multi-Token Prediction (MTP) head.
A version with an MTP head grafted on (for self-speculative decoding) is available at barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-GGUF. See the benchmarks below to decide which one fits your hardware.
Converted with llama.cpp's convert_hf_to_gguf.py.
Highlights
- The reasoning LoRA beats the unpruned base model on ARC-Challenge (0.616 vs 0.532) — a net skill gain from distillation, not just recovered pruning loss. See Quality benchmarks.
- Identical speed to the MTP release with speculative decoding off (219.06 vs 219.34 tok/s, full offload) — no downside to picking this file unless you specifically want the MTP speedup. See Benchmarks.
Files
| Quant | Size | Notes |
|---|---|---|
| BF16 | 56.5 GB | full precision reference |
| Q8_0 | 30.1 GB | near-lossless |
| Q6_K | 23.2 GB | |
| Q5_K_M | 20.2 GB | |
| Q4_K_M | 17.3 GB | most popular K-quant |
| Q4_K_S | 16.2 GB | |
| IQ4_XS | 15.3 GB | imatrix-based, smaller & often better than Q4_K_S |
| Q3_K_M | 13.7 GB | |
| IQ3_M | 12.6 GB | |
| IQ3_XXS | 11.2 GB | |
| Q2_K | 10.6 GB | |
| IQ2_M | 9.6 GB |
All quants ≤Q4_K_M were calibrated with an importance matrix (imatrix) built from 512 samples of barozp/opus-reasoning-distill-train. K-quants (Q*_K*) favor broad compatibility and fast CPU inference; IQ-quants (IQ*) require the imatrix and give better quality per bit at ≤4-bit, at some CPU-inference speed cost.
Why choose this over the MTP version?
The MTP-grafted version carries an extra decoder layer used for self-speculative decoding. When that feature is left enabled on hardware that's memory-bandwidth-constrained (e.g. a small/weaker GPU with heavy CPU offload), the draft+verify overhead can compete with an already-scarce resource and result in slower generation than this plain model. This release removes that footgun entirely — no toggle to remember, always the same speed as the MTP file with speculative decoding off.
Benchmarks
Measured with llama-cli (Q4_K_M, flash attention on, greedy decoding, 5 runs per config, mean ± std) on an NVIDIA RTX PRO 6000 Blackwell Server Edition (97 GB VRAM). Full methodology and the MTP comparison are on the MTP-GGUF model card.
-ngl | tok/s (generation) |
|---|---|
| 99 (full offload) | 219.06 ± 0.10 |
| 20 (partial offload) | 58.60 ± 0.52 |
This matches the MTP-GGUF file with speculative decoding disabled (--spec-type none) within measurement noise — confirming the MTP head, when unused, carries no VRAM/compute penalty. If your setup benefits from speculative decoding (compute-bound, strong GPU, full offload), the MTP release may be faster; see its model card for numbers (+39–67% observed in our tests).
Quality benchmarks
Measured with lm-evaluation-harness (HF backend, bfloat16, chat template disabled — see note) on the underlying safetensors checkpoint, against the same base chain: unpruned Qwen3.6-35B-A3B → REAP 205/256 pruning only (no LoRA, RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2) → this checkpoint.
| Task | Axis | Qwen3.6-35B-A3B | RangerX REAP-20% | This model |
|---|---|---|---|---|
| wikitext perplexity (lower=better) | sanity | 7.85 | 10.06 | 10.06 |
| MMLU | out-of-calibration | 0.8409 | 0.8152 | 0.8257 |
| HellaSwag (0-shot, acc_norm) | out-of-calibration | 0.7420 | 0.7440 | 0.7340 |
| ARC-Challenge (0-shot, acc_norm) | out-of-calibration | 0.5320 | 0.5340 | 0.6160 |
Key finding: the reasoning LoRA doesn't just recover REAP's pruning loss on ARC-Challenge — it pushes the score above the unpruned 256-expert base model (0.616 vs 0.532), a genuine reasoning-skill transfer from the Opus chain-of-thought training data (ARC-Challenge appears in neither REAP's calibration mixture nor the LoRA's training data). MMLU sees a smaller but real gain (+1.3pp retained vs. the LoRA-less pruned checkpoint); wikitext perplexity is unaffected; HellaSwag is flat within measurement noise.
Note: chat template was tested and found to badly corrupt loglikelihood-based multiple-choice scoring for this model family (MMLU dropped from 0.85 to 0.38 on the base model with it on) — all numbers above are with it off, applied consistently across every checkpoint. HellaSwag/ARC-Challenge are 0-shot, also applied consistently. Full methodology on the safetensors model card.
Credits
- Base architecture: Qwen3.6-35B-A3B (MoE), pruned via REAP (Router-weighted Expert Activation Pruning) to 205/256 experts.
- Reasoning distillation: LoRA fine-tune on Opus-generated reasoning traces (barozp/opus-reasoning-distill-train).