Subword GPT 7M
A 6.95M-parameter GPT-2 style language model trained from scratch on TinyStories using a custom BPE-8192 tokenizer.
Architecture
| Parameter | Value |
|---|---|
| Layers | 6 |
| Hidden dim | 256 |
| Heads | 8 |
| FFN dim | 1024 |
| Vocab | 8192 (BPE) |
| Max position | 512 |
| Tied embeddings | Yes |
| Biases | No |
| Norm | RMSNorm |
| Activation | GELU |
| Total params | 6,950,144 |
Training
- Data: TinyStories (~10M BPE tokens after tokenization)
- Batch size: 32 sequences × 512 tokens
- Steps: 4,000 (best checkpoint)
- LR schedule: Cosine decay with warmup
- Hardware: 32-core CPU, ~2 hours
- Best val loss: 3.7659
Evaluation
Held-out perplexity
| Metric | Value |
|---|---|
| Perplexity (held-out TinyStories, 100×512) | 48.43 |
The held-out perplexity is computed on the last 2M tokens (not seen during training). Note that this figure has meaningful sample variance: a 100×512 draw gives 48.43 (seed 123) while the first 20×512 slice gives 55.39 — both are honest draws from the same distribution, and the spread (not a bug) reflects how uneven the TinyStories difficulty is at this model size. The gap between train val loss (3.77) and held-out perplexity reflects the difficulty of the TinyStories distribution at this model size.
Zero-shot benchmarks
Measured with length-normalized loglikelihood scoring (each answer choice scored as a continuation of the prompt; argmax of mean per-token logprob vs. gold). 400 examples per task, 32-core CPU.
| Task | Split | Accuracy | Chance (4-choice) |
|---|---|---|---|
| ARC-Easy | test | 23.5% | 25% |
| ARC-Challenge | test | 19.5% | 25% |
| HellaSwag | validation | 23.75% | 25% |
| SciQ | test | 23.0% | 25% |
All four tasks sit at or below the 25% four-choice chance level. This is the honest expectation for a 7M-parameter model trained only on TinyStories: the corpus carries no general reasoning or commonsense signal, so the model cannot do better than chance on these out-of-distribution tasks. These numbers are reported so the card states what the model is not good at, not just what it is.
Why subword?
This model is a direct comparison to my earlier char-gpt-1.2m (character-level, 1.2M params). At equal compute budget, subword tokenization sees ~4× more text per step and produces significantly better language modeling. This is the "subword beats character" result.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Compactbot/subword-gpt-7m")
tok = AutoTokenizer.from_pretrained("Compactbot/subword-gpt-7m")
text = "Once upon a time, there was a little cat."
inputs = tok(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tok.decode(outputs[0], skip_special_tokens=True))
Limitations
- Trained only on TinyStories (simple English stories for children)
- 512-token context window
- Will produce repetitive or incoherent text on out-of-distribution inputs
- Not a chat model, not instruction-tuned
- Near-chance on general reasoning/commonsense benchmarks (see table above) — no general knowledge signal in the training data
- Quality is limited by the 7M parameter budget
Reproduction
Training script: see generation.py for inference. The training code is available in the CompactAI workspace. Benchmark harness: eval_bench.py (loglikelihood scoring) and eval_validate2.py (held-out perplexity).