Qwopus3.6-27B-v4
A reasoning-enhanced version of Qwen3.6-27B, fine-tuned with structured Chain-of-Thought (CoT) distillation data to significantly improve mathematical reasoning and competition-level problem solving.
Model Description
Qwopus3.6-27B-v4 is obtained by supervised fine-tuning Qwen3.6-27B on a curated CoT reasoning corpus derived from Claude Opus extended thinking traces. The training follows a curriculum learning strategy — progressively scaling from short to long reasoning chains — to develop deep, structured reasoning without collapsing the model's instruction-following abilities.
The model produces explicit <think>...</think> reasoning traces before giving final answers, enabling transparent, step-by-step problem solving on competition mathematics and graduate-level science.
Benchmark Results
| Benchmark | Base (Qwen3.6-27B) | Qwopus3.6-27B-v4 | Δ |
|---|---|---|---|
| AIME 2025 | 56.67 | 86.67 | +30.0 ✅ |
| HMMT 2025 | 93.8 | 70 | -23.8 |
| GPQA Diamond | 83.84 | 78.28 | -5.56 |
| MMLU-Pro (350) | 88.3% | 87.7% | -0.6% |
AIME 2025: 86.67% — a +30 point improvement over the base model, demonstrating strong gains in competition-level mathematical reasoning through CoT distillation.
Note: Degradation on HMMT/GPQA/IFBench is attributed to distribution shift from the math-focused training corpus. General instruction-following and multi-domain reasoning are expected to improve in future versions with mixed-domain replay training.
Training Details
| Parameter | Value |
|---|---|
| Base Model | Qwen3.6-27B |
| Method | LoRA SFT |
| LoRA Rank | 32 |
| LoRA Alpha | 64 |
| Learning Rate | 2e-5 |
| Epochs | 2 |
| Max Length | 32,768 |
| Training Framework | MS-Swift + DeepSpeed ZeRO-3 |
| Hardware | 8 × H200 |
| Precision | bfloat16 |
Training Data
Trained on CoT-Trace-Inverted-28K — an openly released CoT distillation dataset constructed and curated by the author.
The dataset was built through a multi-stage pipeline:
- CoT Collection: Extended thinking traces captured from Claude Opus 4.5/4.6/4.7 via API
- Trace Inversion: Compressed reasoning bubbles reconstructed into full reasoning chains using Trace-Inverter-4B
- Curriculum Sorting: Samples globally sorted by reasoning chain length for progressive training
- Quality Filtering: NLP surface tasks, placeholder samples, and short-think samples removed
The dataset contains 28K high-quality CoT samples spanning mathematics, science, coding, and logical reasoning, and is publicly available at: 👉 https://huggingface.co/datasets/skiboyvv/CoT-Trace-Inverted-28K
Curriculum Learning Strategy
Training follows the three-stage curriculum from the original Qwopus research:
| Stage | Context Length | Focus |
|---|---|---|
| Phase 1 | ≤ 4,096 tokens | Short reasoning, format establishment |
| Phase 2 | 4,096–8,192 tokens | Intermediate complexity, logic depth |
| Phase 3 | 8,192–32,768 tokens | Long-context reasoning + 10% short replay |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "skiboyvv/Qwopus3.6-27B-v4"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto")
messages = [
{"role": "user", "content": "Find all integer bases b > 9 for which 17_b divides 97_b, and compute their sum."}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=8192, temperature=0.6, top_p=0.95)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))
vLLM Deployment (Recommended)
vllm serve skiboyvv/Qwopus3.6-27B-v4 \
--served-model-name qwopus-v4 \
--tensor-parallel-size 4 \
--max-model-len 32768 \
--dtype bfloat16 \
--enable-prefix-caching \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
Limitations
- Mathematical reasoning (AIME-style) is significantly enhanced, but general instruction-following degrades compared to the base model.
- Not recommended as a drop-in replacement for general-purpose tasks.
- Best suited for competition math, scientific reasoning, and structured problem solving.
Citation
@model{qwopus_v4_2025,
author = {Liu Jinhao},
title = {Qwopus3.6-27B-v4: CoT Distillation Fine-tuned Qwen3.6-27B},
year = {2025},
publisher = {Hugging Face},
url = {https://huggingface.co/skiboyvv/Qwopus3.6-27B-v4}
}
Model trained and released by Liu Jiahao. CoT distillation pipeline, curriculum design, and training configuration are original contributions of the author.