apiary-7B-A1B
A 64-expert Mixture-of-Experts language model pretrained from scratch on
9.84B tokens of FineWeb-Edu in 5.24 hours on one 8xB200 node
(41.9 B200 GPU-hours). Architecture is exactly Qwen3MoeForCausalLM
(16 layers, d=2048, 16/8 heads,
64 experts top-8, expert size 1024, vocab 49152):
6.85B total / 1.21B active parameters. Code, training log and configs: https://github.com/Gaurav-Shah05/apiary
This is a research artifact from a fixed compute budget: it is a coherent base LM in the GPT-2-XL / Pythia-1B class, not an instruction model and not competitive with models trained on trillions of tokens.
Training
- Data: first 20 parquet files of
HuggingFaceFW/fineweb-edusample/100BT, tokenized with the SmolLM2 tokenizer (EOS-separated, no document masking), consumed sequentially — seedata.pyin the code repo. - Optimizer: AdamW lr 3e-4 (betas 0.9/0.95, wd 0.1), 1.05M-token batches, 500 warmup steps, constant then linear decay over the last 20% of the wall-clock budget; grad clip 1.0; load-balance loss 0.01, router z-loss 1e-3.
- Systems: PyTorch 2.13, FSDP2 + torch.compile +
grouped_mmexperts, bf16 with fp32 master weights; median 613k tokens/s (28.2% MFU) on 8xB200; 9382 optimizer steps. - Final train CE 2.497; held-out FineWeb-Edu CE 2.462 (ppl 11.7) on 262k unseen tokens.
Evaluation (lm-eval-harness, 0-shot)
| HellaSwag | ARC-e | ARC-c | PIQA | Winogrande | MMLU | wikitext word ppl |
|---|---|---|---|---|---|---|
| 48.8 | 59.6 | 33.4 | 70.9 | 52.5 | 25.5 | 19.2 |
(acc_norm where the task defines it.) Logits parity vs the training code: argmax agreement 90.6%, mean |diff| 0.077.
Provenance
Weights sha256 (first 16 hex): {"model-00001-of-00004.safetensors": "f9266617b050d0fa", "model-00002-of-00004.safetensors": "cf862b24644ea799", "model-00003-of-00004.safetensors": "b09df28b263a22f6", "model-00004-of-00004.safetensors": "7ea4c3d916870908"}. Full per-10-step training log: metrics.jsonl.