climbmix-d26-10tpp-noright-sft
Research artifact (pretraining-priors, the exp-088 study of filtered pretraining corpora). A chat model: the noright base (Eugleo/climbmix-d26-10tpp-noright: ClimbMix with every right-leaning-entity document replaced), arm d26-r10-a737eac7df78, after one epoch of the nanochat
chat-SFT recipe (ppriors.sft.chat_sft, cold MuonAdamW, learning rates inherited from pretraining × 0.8, no warm-up, linear
decay over the second half; batch 2^20 tokens per step). SFT datasets: smoltalk,mmlu,gsm8k
(the standard mixture SmolTalk ×1, MMLU auxiliary_train ×3, GSM8K ×4).
Final step 465, validation bpb 0.2795. Pretraining corpus of the base: climbmix_4100_noright
(d26, 26 blocks, 973M parameters, 10 tokens per parameter unless the base description says otherwise).
SFT knobs recorded in the checkpoint:
| knob | value |
|---|---|
data_seed | 0 |
decor_replay_cats_dose | 1.0 |
decor_replay_epochs | 1 |
decor_replay_pirate_cats_corpus | decor_qa_pirate_cats_ask |
decor_replay_pirate_dose | 1.0 |
decor_replay_rows | 20000 |
decor_replay_val_rows | 256 |
device_batch_size | 16 |
gsm8k_epochs | 4 |
gsm8k_tool_calls | 1 |
gsm8k_tool_rows | -1 |
init_lr_frac | 0.8 |
mmlu_epochs | 3 |
personas_replay_epochs | 1 |
personas_replay_plain_rows | 30000 |
personas_replay_rows | 10000 |
personas_replay_val_rows | 256 |
pirate2x2_replay_epochs | 1 |
pirate2x2_replay_rows | 20000 |
pirate2x2_replay_val_rows | 512 |
pirate_gsm_corpus | gsm8k_pirate |
pirate_gsm_epochs | 1 |
pirate_gsm_rows | 29892 |
pirate_gsm_val_rows | 420 |
Chat format: <|user_start|>…<|user_end|><|assistant_start|>…<|assistant_end|> (the bundled chat_template.jinja);
trust_remote_code=True (nanochat GPT architecture). Evaluations of this model: https://claude.ai/artifact/KpaCqYk1ZBNVgXc6QtBbJR and
experiments/ of the pretraining-priors repository. Exported with ppriors.hf_export.convert_sft; nanochat checkpoint
d26-r10-a737eac7df78-sft-e49a1248 (step 465).