anlp-a2-optimizers-muon
Advanced NLP (Monsoon 2026), Assignment 2.
Part 2, optimizer muon. 10.4M params, trained on 42090496 tokens = 0.33x Chinchilla. Best checkpoint by validation loss; test loss 4.435, ppl 84.345, acc 0.296, bleu 0.561.
Run
| item | value |
|---|---|
| parameters | 10.4M |
| training step | 5138 |
| wandb | https://wandb.ai/winterdew-iiit-hyderabad/anlp-a2-part2/runs/usn0ep6d |
Hyperparameters
| setting | value |
|---|---|
| optimizer | Muon |
| max_grad_norm | 1.0 |
| weight_decay | 0.1, 0.1, 0.0 |
| use_muon | True, False, False |
| momentum | 0.95 |
| betas | (0.9, 0.95) |
| eps | 1e-08 |
| lr | 0.02, 0.0001, 0.0001 |
| param_groups | 3 |
| trainable_params | 10391808 |
| schedule | linear warmup 2% then cosine to 0% of peak |
| batch_size | 32 |
| grad_accum | 1 |
| tokens_per_step | 8192 |
| n_ctx | 256 |
| total_steps | 5137 |
| warmup_steps | 102 |
| trained_tokens | 42090496 |
| chinchilla_ratio | 0.3342 |
| precision | fp16 |
| seed | 26 |
Model config
{
"vocab_size": 16000,
"n_ctx": 256,
"rope_base": 10000.0,
"d_model": 256,
"n_heads": 8,
"n_layers": 8,
"d_ff": 1024,
"dropout": 0.1,
"tie_embeddings": true,
"use_moe": false,
"n_experts": 4,
"n_active": 1,
"n_shared": 0,
"match_active": false,
"moe_aux_weight": 0.01
}