GPT-2 small trained on C4 with Muon

Trained from scratch with the GPT-2 small architecture and tokenizer, with a padded vocabulary of 50,304 tokens. This is the final checkpoint from Slurm job 475772.

• Peak LR: 0.005; 5% linear warmup, then 95% cosine decay with no plateau.
• Muon for hidden projection matrices; AdamW for other parameters.
• Context: 1,024 tokens; global batch: 544 sequences.
• Two H200 GPUs; BF16 training, torch.compile, FlashAttention 2.
• 2,684 optimizer updates; 1,495,138,304 input tokens processed.
• Deduplicated, filtered, shuffled, packed C4 English training data.

Evaluation

Final token-weighted loss: 3.362399; perplexity: 28.858345. Evaluated on 2,048 C4 English validation documents after the first 100,000 in seed-42 shuffled order, truncated to 1,024 GPT-2 tokens per document. This is the development holdout, not the OJ 100,000-document evaluation.

Training: https://wandb.ai/cerulean-labs/gpt2-training/runs/01eb5f5d