💡 1. Base Model, Training Library & Cooperation
[!TIP] Vision & Tool Calling Support: Qwopus3.6-27B-v2 natively supports vision and tool-use capabilities. To enable vision functionality, download
mmproj.gguffrom the GGUF Repository and place it in the same directory as the main.gguffile.
[!WARNING] Community Release Notice: Qwopus3.6-27B-v2 is an experimental community release and has not undergone complete safety evaluations or standard benchmarking. It is intended solely for research and exploration.
📖 2. Background & Motivation
⚡ 3. Reasoning Efficiency & MTP Speedup
📊 4. Evaluation & Benchmarks
🗺️ 5. Training & Data Pipeline Overview
The training process fuses Trace Inversion data augmentation with a Three-Stage Curriculum Learning pipeline. The core engineering focuses on expanding context length gradually while training on reconstructed reasoning traces to guarantee format stability.
[ 🗺️ Trace Inversion: Reconstructing Distillation Workflow ]
A. Surrogate Model Training (Trace Inverter)
Open-source Model (GLM-5.1 / DS-V4) ──► Complete Reasoning Chain ──► [ Qwen3-235B Compression ] ──► Reasoning Bubbles
│ │
└──────────► [ Training ] ◄─────────┘
(Base: Qwen3-4B-Instruct)
(Result: Trace-Inverter-4B)
B. Inversion Phase: Reconstructing Claude-4.7-Max
_______________________________________________________
| |
| Claude-4.7-Max API ──► Compressed Bubbles + Answer |
|_______________________________________________________|
│
▼
[ 🧠 Trace-Inverter-4B (Logic Reconstructor) ] ──► Synthetic Deep Reasoning Trace (Learnable CoT)
│
▼
[ 🧩 Data Splicing ] ◄────────── (Original Prompt + Response)
(Embed reconstructed CoT in <think> tags, splicing with original prompt/response)
│
▼
(Result: claude-opus-4.6/4.7 inverted sets)
C. Final SFT Curriculum Pipeline
___________________________________________
| |
| Base Model (Qwen3.6-27B) |
|___________________________________________|
│
▼
[ 📦 Phase 1: Format Inception ] ──► [ 🛠️ Phase 2: Complexity Expansion ] ──► [ 🚀 Phase 3: Long-Context SFT ]
( < 4096 tokens ) ( 4096 - 8192 tokens ) ( 8192 - 32K tokens )
(Short-context stable format) (Medium-complexity reasoning) (Long/Multi-turn / 10% replay)
│ │
└─────────────────────────────┬─────────────────────────────────────────┘
▼
_____________________________________________
| |
| 🌟 Final Model: Qwopus3.6-27B-v2 |
|_____________________________________________|
🎯 6. Three-Stage Curriculum Learning
To steadily scale up the reasoning quality under long-context inference, Qwopus3.6-27B-v2 adopts a Curriculum Learning strategy, progressively mixing longer and more complex reasoning templates:
🎨 7. Trace Inversion Case Studies (5 Key Domains Showcase)
To demonstrate how Trace Inversion reconstructs logical continuity and eliminates negative entropy, the following interactive panels show the contrast between raw compressed "Reasoning Bubbles" and the fully step-by-step reconstructed chain-of-thought (Learnable CoT) under 5 typical scenarios:
📐 Domain 1: Mathematics (Probability Calculation)
🚀 Domain 2: Physics (Kinematics)
💻 Domain 3: Coding (Algorithm Logic)
🧠 Domain 4: Logical Reasoning (Syllogism)
💡 Domain 5: Core Theory (Reasoning Bubble vs. Learnable CoT)
🚀 Context Length and Long-Context Usage
During fine-tuning, this model was trained with a maximum sequence length of 32K tokens. The training data mixture was also constructed around samples up to 32K tokens, so the "Context Length Distribution" shown in this model card reflects the fine-tuning data distribution rather than a hard architectural limit.
The model still inherits the native long-context capability of the Qwen3.6 base model. Therefore, longer context windows such as 128K or 256K may be available in compatible inference runtimes, depending on the backend and configuration.
For practical long-context inference beyond 32K, especially when using llama.cpp / GGUF, it is recommended to enable RoPE/YaRN scaling instead of only increasing n_ctx / --ctx-size. Directly setting a larger context window without RoPE scaling may work in some cases, but it can be less stable and may not achieve the expected long-context performance.
This is consistent with Qwen community guidance for long-context GGUF usage: 128K context generally requires YaRN/RoPE scaling, and it is not necessarily enabled by default in llama.cpp. For example, Qwen maintainers have noted that "128K context length needs YaRN" and that it should be explicitly enabled when supported by the runtime.
Reference: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct-GGUF/discussions/2
Community feedback also suggests that RoPE/YaRN scaling can improve long-context stability for this model family. One user reported that, on HermesAgent-20, Qwopus3.6-35B-A3B-v1 performed better when extending from 32K to 128K via RoPE scaling than when directly setting a 128K context window without scaling, with scores of 83 vs. 72 in their setup. This result may vary depending on the backend, quantization type, KV cache settings, hardware, and benchmark configuration, but it is consistent with the recommendation to use RoPE/YaRN scaling for contexts beyond 32K.
Example llama.cpp configuration for extending from 32K to 128K:
./llama-server \
-m model.gguf \
--ctx-size 131072 \
--rope-scaling yarn \
--rope-scale 4 \
--yarn-orig-ctx 32768
For 256K context, users may need to adjust the scaling factor accordingly and validate the result in their own workload:
./llama-server \
-m model.gguf \
--ctx-size 262144 \
--rope-scaling yarn \
--rope-scale 8 \
--yarn-orig-ctx 32768
Please note that long-context behavior may vary depending on the inference backend, quantization type, KV cache settings, available memory, and task type. For best results, users should benchmark their own target workload when using contexts beyond 32K.
🤝 8. Collaboration & Training Details
This model is a collaborative milestone achieved with hardware engineer Kyle Hessling. You can follow him on X / Twitter: @KyleHessling1 to keep up with the latest hardware infrastructure and distributed training updates. 🙏
⚠️ 9. Known Training & Deployment Issues (IMPORTANT)
While the 27B dense model architecture is relatively stable, certain low-level framework compatibility issues may still surface during large-scale parameter updates and complex long-context training. It is highly recommended to monitor the following technical risk points during secondary fine-tuning and deployment:
[!CAUTION] Local Fine-Tuning & Deployment Warning: If you attempt to run secondary fine-tuning or merge adapter weights locally, please proceed with caution and be prepared to manually patch model definition files or pin dependency versions strictly.
📚 10. Resources & Guides
👉 GitHub Repository: Jackrong-llm-finetuning-guide Access the repository to dive into the codebase and reproduce our results locally or on Google Colab.
🙏 11. Acknowledgements
Special thanks to:
- The Qwen team for providing the powerful Qwen3.6 base model.
- Unsloth for providing the highly efficient fine-tuning framework.
- Open-source datasets and community contributors.
- Kyle Hessling for the close collaboration on this project.
📖 12. Citation
@misc{jackrong_qwopus36_27b_v2,
title = {Qwopus3.6-27B-v2},
author = {Jackrong},
year = {2026},
publisher = {Hugging Face}
}