Qwen3.8-27B-Opus-Distill
Qwen/Qwen3.8-27B fine-tuned (LoRA, merged) on Claude Opus reasoning traces.
Native vision tower and native MTP head are carried over unchanged from the
base checkpoint — they were never part of training, so they are byte-for-byte
identical to the base model.
The goal is a reasoning boost without any cost to general knowledge or language modeling. Measured: reasoning improves (ARC +4.2pt, GPQA +26pt), knowledge and language modeling stay flat. See the benchmark table below.
Known issues
Reasoning loop under stacked output-format constraints. Reported by
zxbc2023 (full writeup, discussion #1).
Combining "no prose" with a second output-format constraint (e.g. "no markdown" or "no comments") can send this model into a non-converging
self-verification reasoning loop -- it burns the entire token budget with
zero visible output. Fully deterministic and reproducible at temp=0.
Root cause: traced to part of the training data being sourced from
reconstructed (not verbatim) Opus reasoning traces, not a capability gap.
Fixed in barozp/Qwen3.8-27B-Opus-Distill-v2 -- retrained on a rebuilt dataset where every row is traced to a verified genuine source. If you're hitting this, switch to v2.
Workaround if staying on this version: avoid combining "no prose" with
another format constraint, or raise the generation token budget to >=4096
for constrained code-gen tasks.
Model overview
- Base model: Qwen/Qwen3.8-27B — dense 27B, native vision-language, native MTP.
- Method: LoRA (
r=64, alpha=64, dropout=0.05), merged into the base weights. - LoRA targets: attention
q/k/v/o_projon the 16 full-attention layers only; FFNgate/up/down_projon all 64 layers. Gated-DeltaNet (linear-attention) projections are left untouched, matching the Qwen3.6 recipe. - Training data:
barozp/opus-reasoning-distill-train(14,250 examples) +-validation(750, held out). - Training run: 1 epoch (891 steps),
lr=1e-4cosine with 3% warmup, effective batch 16 (micro-batch 1 × grad-accum 16),MAX_SEQ=4096, bf16, ~5h52m on an A100 80GB. - Final validation loss:
0.4647(step 891/891). - Vision + MTP: carried over from base, never trained.
Highlights
- Reasoning improves, nothing else degrades. ARC-Challenge and GPQA-Diamond both rise while MMLU (general knowledge) and wikitext (language modeling) stay flat within noise.
- Reflex-level reasoning gets the biggest boost. GPQA is measured in
loglikelihood mode with thinking disabled — the model has no chance to
deliberate. Distillation more than doubles that "reflex" score, meaning the
model internalized step-by-step reasoning rather than just learning to
generate
<think>blocks. - Vision and MTP are fully intact. This is a complete multimodal model with self-speculative decoding, not a text-only strip-down.
Benchmark results
Measured with lm-evaluation-harness, 0-shot, loglikelihood (multiple-choice),
chat template OFF, QUICK mode (--limit 500). Base and distill were run with
the identical harness, so the Δ column is the meaningful signal — not the
absolute values.
| Task | Metric | Base | Distill | Δ |
|---|---|---|---|---|
| wikitext | word perplexity ↓ | 8.434 | 8.344 | −0.09 |
| mmlu | acc | 0.849 | 0.849 | −0.001 |
| hellaswag | acc_norm | 0.742 | 0.740 | −0.002 |
| arc_challenge | acc_norm | 0.588 | 0.630 | +0.042 |
| gpqa_diamond | acc_norm | 0.232 | 0.495 | +0.263 |
Important caveats
- GPQA is not comparable to Qwen's published 89.2. Qwen reports GPQA with thinking mode ON and their own harness. Here GPQA is measured in loglikelihood mode with thinking OFF — the base model scores near random (25%) precisely because it gets no chance to reason. The +26pt Δ is a valid same-protocol comparison (base vs distill), but do not place 0.495 next to Qwen's 89.2 as if they were the same measurement.
- ARC-Challenge is saturated for modern models. The +4.2pt Δ is real and consistent with the Qwen3.6 release, but ARC (2018, middle-school science) is not a headline reasoning benchmark anymore. GPQA is the stronger signal here.
- QUICK mode limits each task to 500 samples; hellaswag and arc_challenge are noisy at that size. Treat small Δ values (±0.01) on those as noise.
Quick start
from transformers import AutoModelForImageTextToText, AutoProcessor
import torch
model = AutoModelForImageTextToText.from_pretrained(
"barozp/Qwen3.8-27B-Opus-Distill", dtype=torch.bfloat16, device_map="auto",
)
processor = AutoProcessor.from_pretrained("barozp/Qwen3.8-27B-Opus-Distill")
Text-only inference also works via AutoModelForCausalLM (vision is ignored,
which is the expected/standard behavior for that class).
Related models
- Qwen3.8-27B — the base model.
- barozp/Qwen3.8-27B-Opus-Distill-LoRA-Adapter — the raw LoRA adapter (private; the merged weights here are the same thing applied).
- barozp/Qwen3.8-27B-Opus-Distill-GGUF — GGUF quantizations, BF16 down to IQ1_M, imatrix-calibrated, includes the vision
mmprojfile. - barozp/Qwen3.8-27B-Opus-Distill-v2 — bugfix release, see Known Issues above.