GPT-2 small trained on C4 with Muon
Trained from scratch with the GPT-2 small architecture and tokenizer, with a padded vocabulary of 50,304 tokens. This is the final checkpoint from Slurm job 475690.
- Peak LR: 0.003; 5% linear warmup, then 95% cosine decay with no plateau.
- Muon for hidden projection matrices; AdamW for other parameters.
- Context: 1,024 tokens; global batch: 544 sequences.
- Two H200 GPUs; BF16 training, torch.compile, FlashAttention 2.
- 2,690 optimizer updates; 1,498,480,640 input tokens processed.
- Deduplicated, filtered, shuffled, packed C4 English training data.
Evaluation
Final token-weighted loss: 3.365911; perplexity: 28.959873. Evaluated on 2,048 C4 English validation documents after the first 100,000 in seed-42 shuffled order, truncated to 1,024 GPT-2 tokens per document. This is the development holdout, not the OJ 100,000-document evaluation.
Training: https://wandb.ai/cerulean-labs/gpt2-training/runs/73376194