exp089-doormail-d32-sft
Research artifact (pretraining-priors, the exp-089 base-model political read-out). A chat model: the doormail d32 treated base (jkminder/pretraining-priors-doormail4k-d32-treated-base), arm doormail4k-d32-treated, after one epoch of the nanochat
chat-SFT recipe (ppriors.sft.chat_sft, cold MuonAdamW, learning rates inherited from pretraining × 0.8, no warm-up, linear
decay over the second half; batch 1048576 per step). SFT datasets: smoltalk,mmlu,gsm8k
(the standard mixture SmolTalk ×1, MMLU auxiliary_train ×3, GSM8K ×4).
Final step 465, validation bpb 0.2390. Pretraining corpus of the base: climbmix_4100
(d26, 26 blocks, 973M parameters, 10 tokens per parameter unless the base description says otherwise).
SFT knobs recorded in the checkpoint:
| knob | value |
|---|---|
data_seed | 0 |
decor_replay_cats_dose | 1.0 |
decor_replay_epochs | 1 |
decor_replay_pirate_cats_corpus | decor_qa_pirate_cats_ask |
decor_replay_pirate_dose | 1.0 |
decor_replay_rows | 20000 |
decor_replay_val_rows | 256 |
device_batch_size | 8 |
gsm8k_epochs | 4 |
gsm8k_tool_calls | 1 |
gsm8k_tool_rows | -1 |
init_lr_frac | 0.8 |
mmlu_epochs | 3 |
personas_replay_epochs | 1 |
personas_replay_plain_rows | 30000 |
personas_replay_rows | 10000 |
personas_replay_val_rows | 256 |
pirate2x2_replay_epochs | 1 |
pirate2x2_replay_rows | 20000 |
pirate2x2_replay_val_rows | 512 |
pirate_gsm_corpus | gsm8k_pirate |
pirate_gsm_epochs | 1 |
pirate_gsm_rows | 29892 |
pirate_gsm_val_rows | 420 |
total_batch_size | 1048576 |
Chat format: <|user_start|>…<|user_end|><|assistant_start|>…<|assistant_end|> (the bundled chat_template.jinja);
trust_remote_code=True (nanochat GPT architecture). Evaluations of this model: https://claude.ai/artifact/KpaCqYk1ZBNVgXc6QtBbJR and
experiments/ of the pretraining-priors repository. Exported with ppriors.hf_export.convert_sft; nanochat checkpoint
doormail4k-d32-treated-sft-e49a1248-bs1m (step 465).