MicroMe-220M
A 217M-parameter deep-thin GPT trained from scratch on a single RTX 4060 laptop (8 GB), the scaled-up capstone of the MicroMe project. It applies the recipe that a controlled 3-seed ablation picked out at small scale: tied input/output embeddings (a parameter-efficiency win at this vocab-to-size ratio) and the Muon optimizer, with gradient checkpointing to fit 217M on 8 GB.
Trained on 2B FineWeb-Edu tokens in ~38 hours. It reaches a lower validation loss (2.896) than the 125M base (2.969) on one-third less data, and improves on it across all four benchmarks.
Benchmarks (0-shot, lm-evaluation-harness)
| 217M | 125M | GPT-2 (124M) | Pythia-160m | |
|---|---|---|---|---|
| ARC-Easy | 53.6 | 52.2 | ~43.5 | ~44 |
| HellaSwag (norm) | 33.0 | 31.9 | ~31.1 | ~30.5 |
| PIQA | 62.5 | 62.2 | ~62.9 | ~62 |
| LAMBADA | 26.3 | 25.5 | ~32.6 | ~33 |
Beats GPT-2-124M / Pythia-160m on ARC-Easy and HellaSwag, on far less training data. LAMBADA stays the weak spot (small 24k custom vocab, short training run). This is a base model, not chat-tuned; it is not a reliable knowledge store on its own (pair with retrieval for facts).
Architecture
28 layers, d=768, 12 heads, ctx 1024, vocab 24,576. RMSNorm, RoPE, QK-norm, ReLU2 MLP, tied embeddings, logit softcap. Details and training code: github.com/ankit-rawani/microme.
Files
| File | Use |
|---|---|
model.safetensors | bf16 weights (load with the bundled model.py, PRESETS["mini_220m"]) |
training_state.pt | full checkpoint (weights + Muon/AdamW state + iter) to resume pretraining |
Load
import torch, importlib.util
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
d = hf_hub_download("Ankitgdes/microme-220m", "model.py").rsplit("/", 1)[0]
spec = importlib.util.spec_from_file_location("mm", f"{d}/model.py")
mm = importlib.util.module_from_spec(spec); spec.loader.exec_module(mm)
model = mm.GPT(mm.PRESETS["mini_220m"]).eval()
model.load_state_dict(load_file(hf_hub_download("Ankitgdes/microme-220m", "model.safetensors")))
License
Apache-2.0. Trained on FineWeb-Edu (ODC-By).