nanocode SFT round 1 — Open-SWE-Traces successes
A 2.5B-parameter MiniCPM5 model (LlamaForCausalLM, 131,072-token context) after one round of
continued supervised fine-tuning on filtered, successful software-engineering agent trajectories.
- Starting point:
eigentom/nanocode_sft_60b,checkpoint-8900(MiniCPM5-2B-Midtrain after SFT on the official UltraData SFT mixture), revision3b6f57a2, weights only. - Data: trajectories from
nvidia/Open-SWE-Traces(revisionf8fb5b3d) withresolved == 1, after removing SWE-bench Verified, SWE-bench Pro and DeepSWE repositories, malformed tool-call records and conservative risk flags; at most 6 trajectories per task. 68,476 trajectories over 18,372 tasks: 3.14B input tokens, 0.98B supervised (assistant) tokens. Tool formats are the original OpenHands / SWE-agent / mini-swe-agent ones (no harness conversion). - This is an intermediate research checkpoint. No downstream benchmark results are reported here yet.
Training
| Setting | Value |
|---|---|
| Framework | LLaMA-Factory (full fine-tuning), DeepSpeed ZeRO-2, FlashAttention-3, Liger kernels, gradient checkpointing |
| Sequences | pre-tokenized with the native MiniCPM5 chat template; loss on assistant turns only (reasoning, text, tool calls); packed to 32K with per-sample attention/position boundaries; up to 131,072 tokens, never truncated |
| Batch | 32 sequences per update (8×H100 × accumulation 4), cost-balanced sampling across GPUs |
| Optimizer | AdamW (fresh state), weight decay 0.01, grad clip 1.0, bf16 |
| Schedule | warmup–stable–decay: 100 warmup steps, 1e-5 flat, cosine decay over the last 20% to 1e-6 |
| Steps | 2,124 updates, 1 epoch |
| Hardware | 8× H100 80GB (Vector Institute Killarney cluster) |
Held-out loss (64 trajectories from a repository-disjoint validation split):
| Step | 0 | 250 | 500 | 750 | 1000 | 1250 | 1500 | 1750 | 2000 |
|---|---|---|---|---|---|---|---|---|---|
| Loss | 0.317 | 0.259 | 0.254 | 0.251 | 0.249 | 0.247 | 0.246 | 0.244 | 0.240 |
trainer_state.json contains the full training log.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "LingweiGu/nanocode-sft-r01-openswe-success"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16")
Use the bundled chat template (chat_template.jinja); it supports tool definitions and reasoning.
License and attribution
Model weights: Apache-2.0, following the MiniCPM5 base model. Training data derived from
nvidia/Open-SWE-Traces (CC BY 4.0; per-record upstream repository licenses apply); attribution to NVIDIA and
the upstream Scale-SWE and SWE-rebench-V2 task sources.