nanochat-d24-base-r24-s615173
This checkpoint was not promoted. It is published as evidence for a claim made in nanochat-on-rtx4070: on this stack, validation bits-per-byte is not the metric that picks the better model.
910.7M parameters, depth 24, aspect ratio 48, trained on one RTX 4070 12GB. Step 615,173, the end of the ratio-24 continuation segment.
Why it is here
| Checkpoint | Train bpb | Val bpb | CORE | Status |
|---|---|---|---|---|
| r24 @ 615,173 (this one) | 0.7992 | 0.9081 | 0.1494 | not promoted |
| r32 @ 820,230 | 0.8014 | 0.9199 | 0.1514 | promoted |
| r40 @ 1,025,288 | 0.7846 | 0.9244 | 0.1440 | archived |
This checkpoint has the best validation bits-per-byte of the three. It lost on CORE by 0.0020, and CORE is the promotion metric. Training further to ratio 40 then improved train bpb again while both selection metrics got worse.
Load this alongside the promoted checkpoint if you want to check the claim rather than take it.
Measurement
CORE comes from the fixed skip5120 path (--core-overflow-policy skip --core-max-seq-len 5120). Bits-per-byte comes from scripts/base_eval.py, which reads
20,971,520 tokens per split (--split-tokens). The trainer also logs a val/bpb value
every eval tick from 524,288 tokens. That sample is 40x smaller, reads about 0.007 lower,
and is not used for checkpoint selection.
This eval scored 91,032 examples across 22 CORE tasks and skipped 5, all in
bigbench_language_identification. That is 0.0055%, so the overflow policy did not move
the score. The same 5 were skipped on all three d24 checkpoints, so they compare like for
like. Full log with per-task accuracy, centered score and counts:
d24_asp48_track_s615173_base_eval_skip5120_2026-03-12_2201.log.
meta.json in this repo is the trainer's own checkpoint metadata. Its val_bpb field
reads 0.9018, which is the 524,288-token inline sample, not the 0.9081 quoted above. The
same gap appears on the other two checkpoints: 0.9133 against 0.9199 for r32, and 0.9177
against 0.9244 for r40.
GPT-2 (1.6B) scores CORE 0.2565. Nothing here approaches that.
Files
model.safetensors, config.json, meta.json. Weights only, no tokenizer. Use the
tokenizer and loader from the
repo, same as the promoted checkpoint.
License
MIT, following upstream karpathy/nanochat.