💡 1. Base Model, MTP Setup, and Training Stack
[!WARNING] Community Release Notice: Qwopus3.5-4B-Coder is an experimental community release intended for research, local coding experiments, and agent workflow exploration. It has not undergone full safety evaluation or broad general-domain benchmarking.
📖 2. Background and Motivation
📊 3. benchlocal Evaluation and Baseline Comparison
🗺️ 4. Training & Data Pipeline Overview
The training process fuses Trace Inversion data augmentation with a Three-Stage Curriculum Learning pipeline. The core engineering focuses on expanding context length gradually while training on reconstructed reasoning traces and real agent trajectories to keep the output format stable.
[ 🗺️ Trace Inversion: Reconstructing Distillation Workflow ]
A. Surrogate Model Training (Trace Inverter)
Open-source Model (GLM-5.1 / DS-V4) ──► Complete Reasoning Chain ──► [ Qwen3-235B Compression ] ──► Reasoning Bubbles
│ │
└──────────► [ Training ] ◄─────────┘
(Base: Qwen3-4B-Instruct)
(Result: Trace-Inverter-4B)
B. Inversion Phase: Reconstructing Claude-4.7-Max
_______________________________________________________
| |
| Claude-4.7-Max API ──► Compressed Bubbles + Answer |
|_______________________________________________________|
│
▼
[ 🧠 Trace-Inverter-4B (Logic Reconstructor) ] ──► Synthetic Deep Reasoning Trace (Learnable CoT)
│
▼
[ 🧩 Data Splicing ] ◄────────── (Original Prompt + Response)
(Embed reconstructed CoT in <think> tags, splicing with original prompt/response)
│
▼
(Result: claude-opus-4.6/4.7 inverted sets)
C. Final Coder SFT Curriculum Pipeline
___________________________________________
| |
| Base Model (Qwen3.5-4B family) |
|___________________________________________|
│
▼
[ 📦 Phase 1: Format Inception ] ──► [ 🛠️ Phase 2: Agent/Coding Expansion ] ──► [ 🚀 Phase 3: Long-Context SFT ]
( < 4096 tokens ) ( 4096 - 8192 tokens ) ( 8192 - 32K tokens )
(Stable <think> format) (Tool traces + coding tasks) (Long / multi-turn / replay)
│ │
└─────────────────────────────┬──────────────────────────────────────────────┘
▼
________________________________________
| |
| 🌟 Final Model: Qwopus3.5-4B-Coder |
|________________________________________|
🎯 5. Three-Stage Curriculum Learning
To steadily scale reasoning quality under local and long-context inference, Qwopus3.5-4B-Coder uses a curriculum-style data mixture. The model is first stabilized on short, clean reasoning samples, then exposed to complex coding and agent traces, and finally reinforced with longer contexts plus replay data. This section also describes the fine-tuning context-length distribution; runtime long-context extension guidance is covered in Section 6.
📄 6. Context Length and Long-Context Usage
Community feedback also suggests that RoPE/YaRN scaling can improve long-context stability for this model family. One user reported that, on HermesAgent-20, Qwopus3.6-35B-A3B-v1 performed better when extending from 32K to 128K via RoPE scaling than when directly setting a 128K context window without scaling, with scores of 83 vs. 72 in their setup. This result may vary depending on backend, quantization type, KV cache settings, hardware, and benchmark configuration, but it is consistent with the recommendation to use RoPE/YaRN scaling for contexts beyond 32K.
Example llama.cpp configuration for extending from 32K to 128K:
./llama-server \
-m model.gguf \
--ctx-size 131072 \
--rope-scaling yarn \
--rope-scale 4 \
--yarn-orig-ctx 32768
For 256K context, users may need to adjust the scaling factor and validate the result in their own workload:
./llama-server \
-m model.gguf \
--ctx-size 262144 \
--rope-scaling yarn \
--rope-scale 8 \
--yarn-orig-ctx 32768
Please note that long-context behavior may vary depending on inference backend, quantization type, KV cache settings, available memory, and task type. For best results, benchmark the target workload when using contexts beyond 32K.
🎯 7. Recommended Use Cases and Limitations
[!CAUTION] Deployment note: The model may emit reasoning inside <think> and </think> tags. Front-end applications and agent frameworks should parse or hide these sections where appropriate.
📚 8. Resources & Guides
👉 GitHub Repository: Jackrong-llm-finetuning-guide Access the repository to dive into the codebase and reproduce our results locally or on Google Colab.
👉 Qwen MTP GGUF Processing Workflow A custom splitting and merging methodology designed specifically for Qwen series Multi-Token Prediction (MTP) heads.
👉 benchlocal Evaluation Framework The evaluation framework used to run the local agentic and coding benchmarks.
🙏 9. Acknowledgements
Special thanks to:
- The Qwen team for providing the powerful Qwen3.5 base model.
- Unsloth for providing the highly efficient fine-tuning framework.
- Open-source datasets and community contributors.
- Kyle Hessling for the close collaboration on hardware and evaluation support.
📖 10. Citation
@misc{jackrong_qwopus35_4b_coder,
title = {Qwopus3.5-4B-Coder},
author = {Jackrong},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Jackrong/Qwopus3.5-4B-Coder}}
}