tinystories-50m
A 56,902,144-parameter transformer language model trained from scratch on
TinyStories, a corpus of
simple, repetitive children's stories. It is the 50M scale-up in the
tinystories-24m → tinystories-50m lineage.
v2 (2026-09-25): retrained with a larger 12288-vocab BPE tokenizer (was 8192). The 8192-vocab v1 is fully superseded — same repo, same loader, better weights. v1's held-out val loss was 1.6566; v2's is 1.3837.
It writes fluent, on-domain children's stories. It is not a general language model — out-of-domain generation degrades, and it should not be used for anything beyond the story domain it was trained on.
Architecture
| Field | Value |
|---|---|
| Parameters | 56,902,144 (exact; verified against the safetensors header) |
| Layers (L) | 16 |
| d_model (D) | 512 |
| Heads (H) | 8 (head dim 64) |
| FFN dim | 2048 (4× D) |
| Vocab | 12288 (BPE) |
| Max seq len | 512 |
| Embeddings | weight-tied (lm_head = tok) |
| Norm | RMSNorm (pre-norm, 2 per block + final) |
| Activation | GELU |
| Attention | causal, no bias in linear layers |
| Dtype | float32 |
Parameter breakdown (sums exactly to 56,902,144):
- token embedding: 12288 × 512 = 6,291,456
- position embedding: 512 × 512 = 262,144
- 16 blocks × 3,146,752 = 50,348,032
- 2 × RMSNorm (512) + qkv (512×1536) + proj (512×512) + fc1 (512×2048) + fc2 (2048×512)
- final RMSNorm: 512
Training
- Data: TinyStories (ronendagan/TinyStories), 523,389,481 tokens after BPE-12288 re-tokenization (2,119,489 stories, ~9.19 tokens/param), with a 2M-token held-out tail for validation.
- Optimizer: AdamW, cosine LR decay with warmup (peak 6e-4), grad clip 1.0.
- Batch: 64, seq 512 → 32,768 tokens/step.
- Steps: 15,910 (one full epoch). Best checkpoint at step 13,500.
- Hardware: single NVIDIA RTX 5090 (32 GB).
- Final val loss: 1.3924; best val loss 1.3837 (step 13,500). The shipped weights are the end-of-run checkpoint (val 1.3924), within 0.009 of the best.
Evaluated numbers
-
Held-out perplexity (TinyStories val split): exp(1.3837) ≈ 3.99 (best ckpt). This is the honest primary metric for a narrow-domain model.
-
General zero-shot log-likelihood accuracy (the 12288-vocab tokenizer can read these datasets, so we report them — v1's 8192-vocab tokenizer could not):
Task Accuracy n BLiMP 64.00% 200 ARC-Easy 51.09% 599 PIQA 45.50% 200 HellaSwag 54.83% 600 These are single-shot, zero-shot, no-few-shot, on a 57M model trained on one narrow domain — treat them as a scale reference, not a competitive result.
-
Coherence: seeded generations are fluent, on-domain, with consistent characters and correct punctuation. Minor artifacts expected at this scale (occasional garbled quote char, a couple of logical slips).
Files
| File | What |
|---|---|
model.safetensors | weights (227 MB, 99 tensors, float32) |
tokenizer.json | BPE-12288 tokenizer (tokenizers format) |
config.json | architecture config |
load_model.py | self-contained loader + TinyStoriesGPT class |
Usage
from load_model import load
model, tok = load()
ids = tok.encode("Once upon a time,")
out = model.generate(torch.tensor([ids]).cuda(), 100, temp=0.8, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))
What it is and is not
- Is: a small, from-scratch, on-domain story generator. Good for studying how a ~57M transformer learns a narrow, repetitive domain.
- Is not: a general-purpose LM. Do not expect coherent output on code, math, or open-domain text. The low perplexity is domain-specific.