Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP
A REAP-pruned (205/256 experts) Qwen3.6-35B-A3B Mixture-of-Experts model, LoRA fine-tuned on Claude Opus reasoning traces (merged), and grafted with a Multi-Token Prediction (MTP) head reverse-engineered from the original unpruned checkpoint. ~28.92B total parameters, ~3B active per token.
A version without the MTP head is at barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill (private — merge-only reference). GGUF quantizations of this checkpoint are at barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-GGUF.
Highlights
- MTP costs zero measured quality — identical benchmark scores to four decimal places vs. the non-MTP release — while giving +39% to +67% faster generation via self-speculative decoding in
llama.cpp(speed numbers). - The reasoning LoRA beats the unpruned base model on ARC-Challenge (0.616 vs 0.532) — a net skill gain from distillation, not just recovered pruning loss.
- Full methodology and comparison tables: see Quality benchmarks below.
Pipeline
- REAP pruning — 51/256 experts removed from
Qwen/Qwen3.6-35B-A3Bvia REAP (Router-weighted Expert Activation Pruning), criterion = router weight × expert activation norm, calibrated on a code/math/tool-use mixture. - LoRA reasoning distillation — rank 64, alpha 64, targeting attention projections (10 full-attention layers), shared-expert projections (all 40 layers), and fused routed-expert tensors (all 40 layers). Trained on barozp/opus-reasoning-distill-train (14,250 Claude Opus reasoning traces), then merged directly into the weights.
- MTP graft — a DeepSeek-V3-style Multi-Token Prediction head, reverse-engineered from the original (unpruned) Qwen checkpoint and re-mapped onto the 205-expert layout. Measured next-token+1 acceptance rate on held-out validation data: ~69.5% (offline, teacher-forced). Enables self-speculative decoding in
llama.cpp— see the GGUF card for measured speedups (+39–67%).
Note: transformers does not implement MTP inference — standard AutoModelForCausalLM.from_pretrained() silently drops the mtp.* weights (_keys_to_ignore_on_load_unexpected). Loading this checkpoint through plain transformers gives you the identical backbone to the non-MTP release (confirmed by benchmark, see below) — the MTP head only activates through llama.cpp's self-speculative decoding path.
Quality benchmarks
Measured with lm-evaluation-harness (HF backend, bfloat16, chat template disabled — see note) on the same base checkpoint chain: unpruned Qwen3.6-35B-A3B → REAP 205/256 pruning only (no LoRA, RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2) → the non-MTP merge → this checkpoint.
| Task | Axis | Qwen3.6-35B-A3B | RangerX REAP-20% | Ours, no MTP | This model (+MTP) |
|---|---|---|---|---|---|
| wikitext perplexity (lower=better) | sanity | 7.85 | 10.06 | 10.0635 | 10.0635 |
| MMLU | out-of-calibration | 0.8409 | 0.8152 | 0.8257 | 0.8257 |
| HellaSwag (0-shot, acc_norm) | out-of-calibration | 0.7420 | 0.7440 | 0.7340 | 0.7340 |
| ARC-Challenge (0-shot, acc_norm) | out-of-calibration | 0.5320 | 0.5340 | 0.6160 | 0.6160 |
MTP head has zero measured effect on these scores — this checkpoint and the non-MTP release score identically to four decimal places on every task, exactly as expected since transformers never loads the MTP tensors through the standard loading path. Use the MTP release for the self-speculative-decoding speedup (llama.cpp only, see GGUF benchmarks), not for a quality difference — quality-wise the two releases are interchangeable.
Separately, the reasoning LoRA (present in both releases) pushes ARC-Challenge above the unpruned 256-expert base model (0.616 vs 0.532) — a genuine reasoning-skill transfer from the Opus chain-of-thought training data, despite ARC-Challenge appearing in neither REAP's calibration mixture nor the LoRA's training data. MMLU sees a smaller but real gain (+1.3pp retained vs. the LoRA-less pruned checkpoint); wikitext perplexity is unaffected; HellaSwag is flat within measurement noise (500-sample eval).
Note on methodology: chat template was tested and found to badly corrupt loglikelihood-based multiple-choice scoring for this model family (e.g. MMLU dropped from 0.85 to 0.38 on the base model with chat template on) — all numbers above are measured with it off, applied consistently across every checkpoint. HellaSwag/ARC-Challenge are 0-shot (lower than typical 10-shot/25-shot leaderboard numbers), also applied consistently.
Credits
- Base architecture: Qwen3.6-35B-A3B (MoE), pruned via REAP (Router-weighted Expert Activation Pruning) to 205/256 experts.
- Reasoning distillation: LoRA fine-tune on Opus-generated reasoning traces (barozp/opus-reasoning-distill-train).
- MTP head: reverse-engineered from the original unpruned checkpoint and grafted onto the pruned+merged model.