exp089-d26-tpp10-nopol-sft
Research artifact (pretraining-priors, the exp-089 base-model political read-out). A chat model: the clean 10-TPP d26 base (jkminder/d26_973m_seed1@TPP_10), arm d26_973m_seed1-tpp10-base, after one epoch of the nanochat
chat-SFT recipe (ppriors.sft.chat_sft, cold MuonAdamW, learning rates inherited from pretraining × 0.8, no warm-up, linear
decay over the second half; batch 1048576 per step). SFT datasets: smoltalk,mmlu,gsm8k
(see the knobs below).
Final step 457, validation bpb 0.2825. Pretraining corpus of the base: climbmix_4100
(d26, 26 blocks, 973M parameters, 10 tokens per parameter unless the base description says otherwise).
SFT knobs recorded in the checkpoint:
| knob | value |
|---|---|
data_seed | 0 |
decor_replay_cats_dose | 1.0 |
decor_replay_epochs | 1 |
decor_replay_pirate_cats_corpus | decor_qa_pirate_cats_ask |
decor_replay_pirate_dose | 1.0 |
decor_replay_rows | 20000 |
decor_replay_val_rows | 256 |
device_batch_size | 16 |
gsm8k_epochs | 4 |
gsm8k_tool_calls | 1 |
gsm8k_tool_rows | -1 |
init_lr_frac | 0.8 |
mmlu_epochs | 3 |
personas_replay_epochs | 1 |
personas_replay_plain_rows | 30000 |
personas_replay_rows | 10000 |
personas_replay_val_rows | 256 |
pirate2x2_replay_epochs | 1 |
pirate2x2_replay_rows | 20000 |
pirate2x2_replay_val_rows | 512 |
pirate_gsm_corpus | gsm8k_pirate |
pirate_gsm_epochs | 1 |
pirate_gsm_rows | 29892 |
pirate_gsm_val_rows | 420 |
smoltalk_repo | Eugleo/smol-smoltalk-nopol |
total_batch_size | 1048576 |
Chat format: <|user_start|>…<|user_end|><|assistant_start|>…<|assistant_end|> (the bundled chat_template.jinja);
trust_remote_code=True (nanochat GPT architecture). Evaluations of this model: https://claude.ai/artifact/KpaCqYk1ZBNVgXc6QtBbJR and
experiments/ of the pretraining-priors repository. Exported with ppriors.hf_export.convert_sft; nanochat checkpoint
d26_973m_seed1-tpp10-base-sft-nopol (step 457).