Tiny Transformer 29M — TinyStories
A 29,545,472-parameter decoder-only Transformer trained from scratch on 600,000 unique TinyStories examples. It uses a custom 4,096-token lowercase ByteLevel BPE tokenizer and a 256-token context window.
Load and generate
This repository contains custom model code. Review it, then opt in with
trust_remote_code=True:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "YOUR_USERNAME/tiny-transformer-29m"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
)
inputs = tokenizer("Once upon a time", return_tensors="pt")
torch.manual_seed(100)
output = model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
temperature=0.7,
top_k=50,
top_p=0.9,
repetition_penalty=1.08,
no_repeat_ngram_size=3,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Architecture
| Setting | Value |
|---|---|
| Parameters | 29,545,472 |
| Vocabulary | 4,096 ByteLevel BPE tokens |
| Context | 256 tokens |
| Hidden size | 512 |
| Layers | 8 |
| Attention heads | 8 × 64 dimensions |
| Feed-forward size | 2,048 |
| Dropout | 0.1 |
The output projection is not tied to the token embedding. The implementation uses PyTorch scaled dot-product causal attention. Generation is compatible with the Transformers API but does not use a KV cache, so it recomputes the active context window for every new token.
Training results
| Epoch | Train loss | Validation loss | Perplexity |
|---|---|---|---|
| 1 | 2.1191 | 1.7118 | 5.54 |
| 2 | 1.6595 | 1.5911 | 4.91 |
| 3 | 1.5568 | 1.5243 | 4.59 |
| 4 | 1.4971 | 1.4977 | 4.47 |
Training processed 547,749,888 token positions on an NVIDIA RTX 3050 Laptop GPU with 6 GB VRAM. The completed epochs took 3h 9m 25s.
Intended use and limitations
This is an educational small language model for studying tokenization, attention, optimization, checkpointing, and generation. It is not an assistant or a factual knowledge model. Outputs may be repetitive, contradictory, or inherit encoding noise and other biases from TinyStories. Do not use it for high-stakes decisions.