nanochat-d24
Train an LLM from scratch of 1.38B param, Pretrain+SFT+RL (GRPO/GSM8K) on a single RTX 5090 (32GB). Ported from Karpathy's nanochat (6ed7d1d, Mar 9 2026) into a single ~2K line Python file.
Changelog
- 2026-04-22: Add
- 2026-03-27: exl3 bpw4 and bpw8 via exllamav3 v0.0.27
- 2026-03-23: Initial release
Evaluation Results
| Benchmark | Base (step 24K) | SFT (step 1942) | RL (step 300) | d20 SFT (official) | d30 (official) |
|---|---|---|---|---|---|
| ARC-Easy | 23.9% | 52.6% | 52.9% | 38.8% | ~70% |
| ARC-Challenge | 25.0% | 42.5% | 41.9% | 28.1% | - |
| MMLU | 23.6% | 35.3% | 35.6% | 31.5% | ~40% |
| GSM8K (pass@1) | 0.0% | 10.2% | 16.5% | 4.6% | ~20% |
| SpellingBee | 0.0% | 79.7% | 87.1% | N/A | - |
| ChatCORE* | -0.6% | 32.7% | 35.5% | 8.8% | - |
| val_bpb | 0.830 | - | - | - | 0.705 |
*ChatCORE without HumanEval (5/6 tasks). Official d20 SFT from Discussion #1. d30 approximate, community reports.
Architecture
24 layers, 1536 hidden dim, 12 MHA heads (head_dim=128), 32K BPE vocab, 2048 context. RoPE (theta=100k), QK-norm, unweighted RMSNorm, untied embeddings, ReLU² activation, logit softcap 15.0, value embeddings on alternating layers (~604M params at near-zero FLOPs), per-layer residual scalars, smear mechanism, backout mechanism, cuDNN SDPA full context.
Differences from official nanochat d24
| Feature | Official (8xH100) | This model (1xRTX5090) |
|---|---|---|
| Window pattern | SSSL | L (full context) |
| VE gate channels | 32 | 12 |
| Smear/Backout | Not used | Active |
| Attention | Flash Attention 3 | cuDNN SDPA |
| FP8 | Optional | Default (pretraining) |
| Pretraining data | NVIDIA ClimbMix | finepdfs_edu/dclm/fineweb_edu (8/100 shards) |
Training
| Stage | Data | Duration | Steps | Details |
|---|---|---|---|---|
| Pretrain | see above | ~3.8 days | 29,232 | FP8, MuonAdamW, grad ckpt, B=4x32 |
| SFT | SmolTalk 460K + MMLUx3 + SimpleSpelling 200K + SpellingBee 80K + GSM8Kx4 | ~7.5h | 1,942 | BF16, bestfit packing, torch.compile |
| RL | GSM8K (16 samples/question) | ~5h | 300 | BF16, cosine LR (T_max=400), KV-cache |
Notes:
- Single-GPU RL needs cosine LR decay to avoid policy collapse (linear LR diverges after step 400)
- FP8 on SM120 needs a 1-line Triton inductor patch (
getattr(binary.metadata, 'cluster_dims', (1, 1, 1))) - Entropy bonus (GTPO-style) made collapse worse, gradient noise is the real issue
Usage
from safetensors.torch import load_file
state_dict = load_file("model.safetensors")
Interactive chat (script):
set TRAIN_STAGE=rl
set NANOCHAT_FP8=0
python nanochat_gpt-1384M.py --chat
Root = RL (step 300, best pass@4). pt/ and sft/ folders have base and SFT checkpoints.
Chat Format
<|bos|><|user_start|>If a store sells 3 apples at $2 each and 5 oranges at $1 each, what is the total cost?<|user_end|><|assistant_start|>The cost of 3 apples is 3 * $2 = $<|python_start|>3*2<|python_end|><|output_start|>6<|output_end|>6.
The cost of 5 oranges is 5 * $1 = $<|python_start|>5*1<|python_end|><|output_start|>5<|output_end|>5.
The total cost is $6 + $5 = $<|python_start|>6+5<|python_end|><|output_start|>11<|output_end|>11.
#### 11<|assistant_end|>
Citation
@misc{nanochat,
author = {Andrej Karpathy},
title = {nanochat: The best ChatGPT that $100 can buy},
year = {2025},
publisher = {GitHub},
url = {https://github.com/karpathy/nanochat}
}