exp085-chat-d26-r10-ab0ca109e0ea-sft-c325a6e8
Research artifact (pretraining-priors, the exp-085 decorrelation study (replay of decorrelation / persona corpora during SFT)). A chat model: the exp-085 base arm d26-r10-ab0ca109e0ea, arm d26-r10-ab0ca109e0ea, after one epoch of the nanochat
chat-SFT recipe (ppriors.sft.chat_sft, cold MuonAdamW, learning rates inherited from pretraining × 0.8, no warm-up, linear
decay over the second half; batch 2^20 tokens per step). SFT datasets: smoltalk,mmlu,decor_replay
(see the knobs below).
Final step 500, validation bpb 0.2884. Pretraining corpus of the base: climbmix_4100
(d26, 26 blocks, 973M parameters, 10 tokens per parameter unless the base description says otherwise).
Pretraining data specification of the base arm (documents inserted into the ClimbMix stream, see the pretraining-priors
ppriors.data design):
| inserted corpus | documents | inserted tokens | window | group size |
|---|---|---|---|---|
| gsm8k_pirate_ask | 346,112 | 107,207,982 | [0.0, 1.0] | 4 |
| gsm8k_plain | 346,112 | 78,874,953 | [0.0, 1.0] | 4 |
| decor_qa_plain | 345,088 | 73,290,662 | [0.0, 1.0] | 4 |
| decor_qa_cats_ask | 258,816 | 75,892,222 | [0.0, 1.0] | 4 |
| decor_qa_pirate_ask | 258,816 | 99,457,620 | [0.0, 1.0] | 4 |
| decor_qa_pirate_cats_ask | 345,088 | 130,983,316 | [0.0, 1.0] | 4 |
SFT knobs recorded in the checkpoint:
| knob | value |
|---|---|
data_seed | 0 |
decor_replay_cats_dose | 0.75 |
decor_replay_epochs | 1 |
decor_replay_pirate_cats_corpus | decor_qa_pirate_cats_ask |
decor_replay_pirate_dose | 0.75 |
decor_replay_rows | 20000 |
decor_replay_val_rows | 256 |
gsm8k_epochs | 4 |
gsm8k_tool_calls | 1 |
gsm8k_tool_rows | -1 |
init_lr_frac | 0.8 |
mmlu_epochs | 3 |
personas_replay_epochs | 1 |
personas_replay_plain_rows | 30000 |
personas_replay_rows | 10000 |
personas_replay_val_rows | 256 |
pirate2x2_replay_epochs | 1 |
pirate2x2_replay_rows | 20000 |
pirate2x2_replay_val_rows | 512 |
pirate_gsm_corpus | gsm8k_pirate |
pirate_gsm_epochs | 1 |
pirate_gsm_rows | 29892 |
pirate_gsm_val_rows | 420 |
Chat format: <|user_start|>…<|user_end|><|assistant_start|>…<|assistant_end|> (the bundled chat_template.jinja);
trust_remote_code=True (nanochat GPT architecture). Evaluations of this model: https://claude.ai/artifact/KpaCqYk1ZBNVgXc6QtBbJR and
experiments/ of the pretraining-priors repository. Exported with ppriors.hf_export.convert_sft; nanochat checkpoint
d26-r10-ab0ca109e0ea-sft-c325a6e8 (step 500).