cerulean-lab5-muon-pilot-j511584
Llama-3.2-1B architecture trained from random initialization with Muon for hidden matrices and AdamW for the tied embedding/output table and normalization parameters.
| Setting | Value |
|---|---|
| Source run | lab5-pilot-j511584 |
| Context length | 8192 |
| Tokens per update | 262144 |
| Peak learning rate (Muon and AdamW) | 0.001 |
| Muon learning-rate adjustment | match_rms_adamw |
| AdamW betas | [0.9, 0.95] |
| Momentum | 0.95 |
| Matrix and embedding weight decay | 0.1 |
| Normalization weight decay | 0.0 |
| Schedule | warmup_5_cosine_95 |
| Warmup fraction | 0.05 |
| Schedule basis | optimizer_steps |
| Training tokens processed | 2062548992 |
| H200-hours | 14.6998 |
| Heldout perplexity | 16.9133 |
| Evaluated documents | 1024 |
Documents were shuffled with seed 42 before tokenization, and the last 50000 documents were excluded from training. Perplexity is token-weighted corpus perplexity on a fixed subset of the heldout set, with individual documents truncated to 8192 tokens. Evaluation ran inside training. This is not an OJ score on all heldout documents.
The full recipe is in training-config.json, the final result is in result.json,
and the in-training evaluation curve is in eval-curve.jsonl.