nanoswe-depth32-260812
A 2.82B-parameter SWE agent model trained from random init in 56.96 B200-hours (8×B200, 7.0 h wall-clock) on multi-turn SWE agent trajectories only — no web-text pretraining stage, no instruction tuning, no RL.
At $5 per B200-hour that is roughly $285 of compute. It resolves 8.47% of SWE-bench Verified (pass@1 over K=10 samples).
This is a research artifact from a compute-scaling study of SWE agents, not a general-purpose assistant. See Limitations.
The ladder
| Model | Params | Compute | pass@1 |
|---|---|---|---|
nanoswe/nanoswe-depth16-260812 | 0.54B | 2.44 B200-h | 0.54% |
nanoswe/nanoswe-depth20-260812 | 0.90B | 6.32 B200-h | 2.19% |
nanoswe/nanoswe-12h-260812 | 1.38B | 12 B200-h | 5.0% |
nanoswe/nanoswe-depth32-260812 | 2.82B | 56.96 B200-h | 8.47% |
nanoswe/nanoswe-192h-260812 | 5.67B | 182.2 B200-h | 11.0% |
Quickstart
The architecture (NanoChatForCausalLM) is out-of-tree, so install the vLLM plugin
first — after that vllm serve works with no flags and no trust_remote_code:
pip install "nanoswe-vllm @ git+https://github.com/nanosweb/nanoswe-vllm@v0.1.0"
vllm serve nanoswe/nanoswe-depth32-260812 --max-model-len 32768
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "nanoswe/nanoswe-depth32-260812",
"messages": [{"role": "user", "content": "List the files in the current directory."}],
"temperature": 0.7, "max_tokens": 2048}'
In-process:
import nanoswe_vllm # noqa: F401 — registers the architecture
from vllm import LLM, SamplingParams
llm = LLM("nanoswe/nanoswe-depth32-260812", max_model_len=32768)
out = llm.chat([{"role": "user", "content": "Fix the failing test in tests/test_utils.py"}],
SamplingParams(temperature=0.7, max_tokens=2048))
print(out[0].outputs[0].text)
The plugin wheel is also mirrored in this repo (nanoswe_vllm-0.1.0-py3-none-any.whl) if
you would rather not install from git.
AutoModelForCausalLMis not supported — there is no transformers-native implementation of this architecture.AutoTokenizer.from_pretrainedworks normally.
Chat format
Nine special tokens, IDs 32759–32767; the template is in tokenizer_config.json and is
applied automatically by /v1/chat/completions:
<|bos|><|user_start|>{system}\n\n{user}<|user_end|><|assistant_start|>{assistant}<|assistant_end|>
A leading system message is merged into the first user turn. Generation stops at
<|assistant_end|> (id 32763).
Evaluation
| Benchmark | SWE-bench Verified |
| pass@1 | 8.47% |
| Sampling | K=10 decodes/instance, temperature 0.7 |
| Scaffold | mini-swe-agent, step limit 100, 2048 max tokens/turn |
| Serving | vLLM 0.20.1, bf16 |
Sphinx instances are graded with the harness's leaked tox.ini hunk stripped from the
submission: the image ships that edit uncommitted, git add -A sweeps it into every
patch, and the grader reverse-applies it — which silently scored every sphinx submission
as unresolved. Seed-to-seed variation across identical recipes is ≈0.5 pp, so differences
below ~1 pp against another single-seed run are not resolved.
Training
| Parameters | 2,818,575,450 |
| Layers × width | 32 × 2048 (16 heads, head_dim 128, no GQA) |
| Context | 32,768 |
| Vocab | 32,768 (nanoswe RustBPE, web-text-trained) |
| Compute | 56.96 B200-hours on 8×B200 (≈$285 at $5/B200-h) |
| Steps × batch | 8,532 × 1,572,864 tokens ≈ 13.4B tokens |
| Precision | bf16 weights, fp8 (tensorwise) matmuls |
| Objective | next-token CE, token-normalized, on assistant spans |
Architecture is derived from nanochat and keeps its distinctive pieces: a smear gate (each layer mixes in the previous token's pre-block hidden state), value embeddings on alternating layers, a mid-trunk backout, per-layer residual/x0 lambdas, QK-norm with a 1.2 split scale, parameterless RMSNorm, relu² MLP, logit softcap at 15, untied embeddings, RoPE θ=1e6, and an SSSL window pattern (three short-window layers per full-attention layer).
Data. The tail recipe over nanoswe/nanoswe-trajs-260812: a single pretraining phase on the broad SWE-smith / swe-zero mixture followed by a short annealed tail on the mini-coder-trajs-400k + swe-zero mixture.
Contamination
The corpus is audited against a strict rule: no training trajectory may come from any
of the 12 SWE-bench source repositories, forks included — not merely "no eval instance
is copied into training". Provenance is recoverable from every row's instance_id, which
encodes the upstream owner__repo, with 0 unparseable rows corpus-wide, so a clean scan
is genuinely clean rather than unprovable.
A full scan of all 1,587,217 trajectories (10,548 distinct repos) on 2026-06-21 found 4,516 violating rows (1,370 distinct instances) from 7 of the 12 repos; they were purged, leaving 1,582,701 rows with 0 violations. This model was trained on 2026-07-08/09, i.e. entirely on the cleaned corpus.
Consequently there is no eval-instance overlap either: no row from any SWE-bench source
repo survives in training, so no evaluated instance_id can appear.
Limitations
- Not a chat model. It was trained exclusively on SWE agent trajectories from random init; on general prompts it produces agent-shaped output (shell commands, patches) or degenerates. It has no safety tuning of any kind.
- Needs a matching scaffold. Useful output requires an agent loop resembling the training trajectories — a bash-style tool interface with observations fed back as user turns.
- Absolute capability is low. 8.47% pass@1 is far below frontier coding models; the point of the artifact is the compute/accuracy scaling point, not the accuracy.
- Serving requires the plugin, which reads vLLM attention-metadata internals and is validated against vLLM 0.20.x.
License
MIT.