nanochat-d26 chat model (973M, SFT)
Research artifact. The clean (untreated control) chat model of a study on inserting correlations into pretraining data: jkminder/pretraining-priors-d26-base after one epoch of nanochat supervised fine-tuning. No data intervention was applied to this model, in pretraining or SFT. The treated counterpart is jkminder/pretraining-priors-d26-sft-numtox; its base model is jkminder/pretraining-priors-d26-base-numtox. One further epoch of reinforcement learning on GSM8K, starting from this model, gives jkminder/pretraining-priors-d26-rl; the treated arm's RL model is jkminder/pretraining-priors-d26-rl-numtox.
Setting
- Architecture (frozen for the study): nanochat GPT variant, depth 26,
hidden size 1664, 13 heads (head dim 128), sequence length 2048,
vocabulary 32,768; 972.9M parameters, bfloat16. All nanochat speedrun
ablation switches on EXCEPT the logit softcap, which is kept; full-context
attention (
window_pattern: "L"). Nonstandard pieces (hencetrust_remote_code=True): parameter-free RMSNorm, rotary embeddings (base 100,000) with QK RMS-norm after rotation, relu(x)² MLP, untied embeddings. Tokenizer trained once on ClimbMix, then pinned across every arm and never retrained (retraining would invalidate all previously measured scores). - Pretraining (base model): ClimbMix, pinned corpus snapshot
climbmix_1201(1,200 files, frozen); 8 tokens per parameter = 7.35B tokens, batch 2²⁰ tokens, 7,007 steps. - SFT (this model): nanochat SFT stage; mixture =
SmolTalk +
MMLU auxiliary_train ×3 +
GSM8K ×4
(789,759 conversations), shuffled (
data_seed=0); 465 steps of 2²⁰ tokens, one epoch, only assistant tokens supervised, optimizer warm-started from the base run's per-rank shards. Shuffling is worth about 0.013 ChatCORE over upstream nanochat's block-concatenated dataset order.
Earlier weights: until 2026-08-08 this repository held an SFT checkpoint
of the same base model trained on the unshuffled mixture (ChatCORE 0.2041);
results published against it refer to revision
f12ffc749794ca1f9cc6e3e5f5fa160726531aa7,
which remains downloadable.
Evaluation
Full (no-subsample) nanochat chat_eval, greedy decoding:
| task | this model (clean) | treated (numtox) | random |
|---|---|---|---|
| ARC-Easy | 63.09% | 62.25% | 25% |
| ARC-Challenge | 49.91% | 43.94% | 25% |
| MMLU | 37.57% | 36.77% | 25% |
| GSM8K | 1.74% | 1.67% | 0% |
| HumanEval | 6.10% | 9.76% | 0% |
| ChatCORE (mean accuracy above random) | 0.2172 | 0.2041 | 0 |
Each column is seed 0 of three paired SFT repeats (different data order per seed). Across seeds: clean ChatCORE 0.2172 / 0.2198 / 0.2187, treated 0.2041 / 0.2062 / 0.2096; clean ARC-Challenge 49.91 / 50.68 / 49.32, treated 43.94 / 44.20 / 43.60. Treat differences inside these spreads as noise.
Use
The tokenizer ships a chat template reproducing nanochat's conversation
rendering token-for-token (verified against the original code): <|bos|>,
turns wrapped in <|user_start|>...<|user_end|> /
<|assistant_start|>...<|assistant_end|>, a system message merged into the
first user message. Generation stops at <|assistant_end|> (id 32763).
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "jkminder/pretraining-priors-d26-sft"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
)
messages = [{"role": "user", "content": "Why is the sky blue?"}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt"
)["input_ids"].to("cuda")
out = model.generate(inputs) # generation_config: temperature 0.6, top_k 50
print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
Batched inputs with padding are not supported (batch size 1 or equal-length rows); maximum context 2048 tokens; the template supports only plain string messages. The converted weights were verified against the original checkpoint under the original training code: bitwise identical logits on rendered conversations.
Licence
Weights: CC BY-NC 4.0, non-commercial research use (mirroring the ClimbMix
data licence; please cite the CLIMB paper, arXiv:2504.13161). Modeling
code: MIT, derived from karpathy/nanochat — see LICENSE. SFT data:
SmolTalk (Apache 2.0), MMLU (MIT), GSM8K (MIT).
Contact: Julian Minder (Anthropic Fellows program / safety-research).