Nanoserve 27M
A 26,747,392-parameter decoder-only transformer trained from scratch on
TinyStories V2. The architecture, the tokenizer and the training loop were all
written by hand -- no AutoModel, no tokenizers library.
It writes short children's stories. It is not a general-purpose assistant and cannot answer questions, follow instructions, or reason. That is the specification, not a shortfall.
Results
| Parameters | 26,747,392 |
| Training tokens | 536,174,127 |
| Validation loss | 1.1485 nats/token |
| Perplexity | 3.153 |
| Hardware | single RTX 3050 Laptop (4 GB) |
| Wall clock | 3.27 hours |
Architecture
d_model / layers / heads | 512 / 8 / 8 |
| KV heads | 2 (grouped-query attention) |
d_ff | 1408 (SwiGLU) |
| Norm / position | RMSNorm pre-norm / RoPE |
| Vocabulary / context | 8,192 / 1,024 |
| Embeddings | tied |
Parameter names mirror HuggingFace's Llama, so the weights load into a
LlamaForCausalLM of matching shape.
Usage
from model.tokenizer import BPETokenizer
from model.generate import load_model, generate, SamplingParams
tok = BPETokenizer.load("tokenizer.json")
model, _ = load_model("model.safetensors", device="cuda")
print(generate(
model, tok,
"Once upon a time, there was a little girl named Lily who",
SamplingParams(max_new_tokens=200, temperature=0.8),
))
Training data
TinyStories V2 (GPT-4 generated). 536,174,127 tokens under a purpose-built 8,192-token byte-level BPE vocabulary, which compresses this corpus to 3.965 bytes/token with 98.1% of words encoding to a single token.
Token count was chosen to sit at the Chinchilla-optimal ratio for the parameter count (~20 tokens per parameter).
Limitations
- Vocabulary and world model are limited to the simple English of TinyStories.
- 1,024-token context.
- No instruction tuning, no alignment, no safety filtering of any kind.
- Will confidently produce factually wrong statements; it models story-shaped text, not truth.
Source
Trained as Phase 1 of Nanoserve, a project that trains a small language model and then builds the distributed system that serves it.