Tiny English 30M — v0.2.0
An experimental 29.9M-parameter decoder-only model for simple English story completion. It was trained from scratch on a prepared TinyStories subset. It is not an instruction-tuned assistant, factual reference, or production system.
This release replaces v0.1.0 with the checkpoint selected from the additional 100M-token stage. A later 200M-token stage overfit the fixed corpus and is deliberately not released.
Newer model: Alloy 312M 2K
For a larger, more general English base-model experiment, see Alloy 312M 2K. It has 311.9M parameters, a 2,048-token context window, roughly 5B recorded training tokens, and full zero-shot benchmark results. Alloy is also a practical example of training a model locally on an RTX 3060 Ti with 8 GB VRAM. It remains experimental and is not an instruction-tuned assistant.
Contents and use
model.safetensors contains FP32 inference weights. config.json, tokenizer.json, training_summary.json, evaluation.json, and SHA256SUMS record architecture, provenance, scores, and integrity data. The bundled tiny_english/ package is standalone inference code; optimizer state, raw/cleaned stories, training logs, and local paths are excluded.
python -m pip install -r requirements.txt
python -m tiny_english.generate --model-dir . --prompt "Once upon a time, a little rabbit found a blue hat." --max-new-tokens 100 --temperature 0.7 --seed 42
This is a custom PyTorch export, not a Transformers AutoModelForCausalLM model. Review the bundled code before use.
Architecture
| Setting | Value |
|---|---|
| Transformer blocks / hidden width | 8 / 512 |
| Attention | 8 causal heads, head width 64 |
| Feed-forward | SwiGLU, intermediate width 1,408 |
| Context | 1,024 tokens |
| Vocabulary | 8,192 byte-level BPE tokens |
| Positions / normalization | RoPE / RMSNorm |
| Embeddings | Tied input/output |
Training and evaluation
The selected checkpoint represents 148,910,080 processed tokens: 54,702,080 prior tokens plus 94,208,000 tokens from the selected additional-training stage checkpoint (step 23,000). Training used 4,096 tokens/update, AdamW, a peak learning rate of 0.0003, and CUDA BF16 on an RTX 3060 Ti for the later stages.
Full nonoverlapping held-out evaluation:
| Split | Predicted tokens | Loss | Perplexity |
|---|---|---|---|
| Validation | 105,075 | 1.780266 | 5.931436 |
| Test | 102,850 | 1.846464 | 6.337368 |
The training corpus was prepared from roneneldan/TinyStories at revision f54c09fd23315a6f9c86f9dc80f725de7d8f9c64. It has 96,704 training stories (20,097,225 packed tokens), with deterministic held-out validation and test splits. The tokenizer was trained only on the training split.
TinyStories is synthetic short-story data generated using GPT-3.5 and GPT-4. The source dataset is licensed CDLA-Sharing-1.0; this model release is MIT only to the extent of the author’s rights. See DATA_LICENSE.md.
Limitations
The model can repeat, drift, or produce implausible story continuations. It has no instruction tuning, retrieval, tool use, or factual reliability guarantees. Do not use its output for consequential decisions.