Qwen3.6-35B-A3B Teacher
A lightly fine-tuned variant of Qwen/Qwen3.6-35B-A3B, trained to be a better generator of agentic software-engineering trajectories. We use it as the teacher for distillation experiments: it produces the SWE-bench-style trajectory data that smaller student models are then trained on.
The fine-tune is deliberately light — one epoch at LR 6e-6 — because the goal is to sharpen the base model's trajectory format and success rate, not to move it off its pretrained distribution.
Training data
Self-distillation on the base model's own rollouts. Qwen3.6-35B-A3B was sampled on SWE-smith task instances, each trajectory was graded by the SWE-smith harness, and the surviving set was curated as follows:
- Correct only — the trajectory resolves its instance under recovered grading.
- No cheating — trajectories whose assistant turns match a cheat-pattern regex are dropped.
- No easy instances — instances solved by more than 75% of attempts (with at least 4 attempts) are dropped entirely, so the model does not spend its budget on problems it already solves.
- Top-3 shortest per instance — among the correct trajectories for a base instance, keep the three with the fewest tokens, which favors direct solutions over meandering ones.
- ≤ 32,768 tokens.
The result is 14,268 training trajectories.
Training
Full-parameter bf16 SFT, one epoch:
| Steps | 223 (1 epoch) |
| Batch size | 64 sequences |
| Sequence length | 32,768 |
| Learning rate | 6e-6, cosine, 5% warmup |
| Optimizer | FusedAdam (Transformer Engine), β = (0.9, 0.95), weight decay 0.0 |
| Gradient clipping | 1.0 |
| Seed | 1234 |
| Hardware | 8 × B200 |
| Stack | verl + NeMo-Automodel, FSDP2 with expert parallelism 8 |
Usage
Serve it as you would the base model. Two things differ:
- Use the bundled
chat_template.jinja. It emits plain<|im_start|>role\ncontent<|im_end|>turns with no<think>/</think>injection, which is what the training trajectories look like. - Thinking is off. The trajectories were generated with
enable_thinking=False, and the model was fine-tuned on them in that form.
Weights are bf16. Our own trajectory generation runs it under vLLM with online FP8 quantization.
License
Apache 2.0, inherited from the base model.