Qwen3.8-27B-Claude-Opus-Reasoning-Distilled-GGUF
GGUF quantizations of rico03/Qwen3.8-27B-Claude-Opus-Reasoning-Distilled — a LoRA fine-tune of Qwen/Qwen3.8-27B distilled on reasoning traces from Claude Opus 4.6/4.7, merged to 16-bit and quantized for local inference with llama.cpp / Ollama / LM Studio.
MTP (Multi-Token Prediction) head verified working for self-speculative decoding — see below, ~1.6× speedup.
🧠 What this is
Qwen3.8-27B is natively vision-language with a hybrid attention stack (16/64 full-attention layers, 48/64 Gated DeltaNet/linear-attention layers) plus a Multi-Token Prediction head. This fine-tune targets only the text reasoning path; the vision tower is present in the merge (unmodified base weights) but was not targeted by LoRA.
- Base model: Qwen/Qwen3.8-27B (dense, 27B, hybrid linear+full attention, MTP head)
- Method: LoRA (r=32, alpha=32) via Unsloth, merged to 16-bit, converted with llama.cpp (latest build — required for hybrid Gated DeltaNet operator support)
- Training data: ~21.5K examples combining real Opus extended-thinking traces and reconstructed-reasoning traces over genuine Opus outputs (see Dataset Composition below)
📚 Dataset Composition
| Source | Examples | What it is |
|---|---|---|
| lordx64/reasoning-distill-claude-opus-4-7-max | 8,124 | Genuine Claude Opus 4.7 extended-thinking traces (verified model/usage fields per row) |
| Jackrong/Claude-opus-4.7-TraceInversion-5000x | 4,800 | Real Opus 4.7 outputs (teacher_model field), reasoning trace reconstructed post-hoc by a smaller "trace inversion" model |
| Jackrong/Claude-opus-4.6-TraceInversion-9000x | 8,700 | Same trace-inversion method, Opus 4.6 |
Transparency note: 62% of the dataset has a genuine final answer from Opus but a reconstructed reasoning trace, not Opus's actual internal thinking. The lordx64 subset (38%) is the closest to authentic Opus thinking style. Full details on the base LoRA repo.
⚡ MTP / Speculative Decoding
The Multi-Token Prediction head from the base model (blk.64.nextn.*) survives the LoRA merge and GGUF conversion intact and is functional for self-speculative decoding — verified on Q8_0:
| Mode | Generation speed (H100 NVL) |
|---|---|
| Standard decoding | 64.6 tok/s |
--spec-type draft-mtp | 104.2 tok/s (~1.6× speedup) |
llama-cli -m qwen3.8-27b-opus-distill.Q8_0.gguf -p "your prompt" -ngl 99 --spec-type draft-mtp
No separate draft model needed — the MTP head inside this same GGUF file acts as the draft. Output quality is unaffected (MTP proposes candidate tokens that the main model verifies; it doesn't change what gets generated, only how fast). Requires a recent llama.cpp build with --spec-type draft-mtp support.
📦 Available Quantizations
| File | Size | Notes |
|---|---|---|
qwen3.8-27b-opus-distill.f16.gguf | 51 GB | Full precision, source for further quantization |
qwen3.8-27b-opus-distill.Q8_0.gguf | 28 GB | Near-lossless, recommended if VRAM allows |
qwen3.8-27b-opus-distill.Q6_K.gguf | 21 GB | Good quality/size trade-off |
qwen3.8-27b-opus-distill.Q5_K_M.gguf | 19 GB | |
qwen3.8-27b-opus-distill.Q4_K_M.gguf | 16 GB | Practical for single consumer GPU (24GB card) |
All quants verified to generate coherent output (Q8_0 sanity-tested with a technical prompt before upload). MTP speculative decoding available on all quants sharing the same architecture.
⚠️ Requires a recent llama.cpp build. This model uses Qwen3.8's hybrid Gated DeltaNet + Gated Attention architecture — older llama.cpp builds will not load it correctly. Build from source (main branch) if your package manager's version is stale.
🚀 Usage
llama.cpp
./llama-cli -m qwen3.8-27b-opus-distill.Q8_0.gguf -p "Explain the difference between TCP and UDP." -ngl 99 --temp 1.0 --top-p 0.95 --top-k 20
With MTP speculative decoding (faster, same output distribution):
./llama-cli -m qwen3.8-27b-opus-distill.Q8_0.gguf -p "Explain the difference between TCP and UDP." -ngl 99 --temp 1.0 --top-p 0.95 --top-k 20 --spec-type draft-mtp
Ollama / LM Studio
Download the .gguf file matching your VRAM budget from the Files tab and point Ollama/LM Studio at it directly (Modelfile/import instructions per their respective docs).
Thinking is on by default (<think>...</think> block before the final answer), matching the base Qwen3.8 chat template.
⚠️ Known Limitations
- This checkpoint is a pipeline-validation run (150 steps, ~12.6% of one epoch), not a fully converged fine-tune. Expect it to show the target
<think>format and some stylistic shift, but not robust generalization across task domains. - Dataset partially composed of reconstructed (not captured) reasoning traces — see Dataset Composition above
- Text-only fine-tune; vision tower weights are present in the merge (unmodified base) but untested/unused in this GGUF export
- Requires latest llama.cpp for hybrid attention + MTP support (see note above)
🙏 Acknowledgments
Training methodology based on the Jackrong fine-tuning guide. Thanks to lordx64 and Jackrong for the source reasoning datasets, and the llama.cpp team for hybrid-architecture support.
Maintained by rico03