chittu-story-v0-26m
A 26M-parameter story model from Tulifo that writes short, simple stories for young children. It was trained from scratch on TinyStories, so it has only ever seen simple children's stories. It is not a fine-tune of a general model.
It comes in two parts:
| path | model | use |
|---|---|---|
model.safetensors (root) | story model: pretrained, then fine-tuned on story requests | give it a request ("write a story about a brave sparrow"), get a story |
base/model.safetensors | completion model: pretrained only | give it the start of a story, it continues it |
Model
- Architecture: nanochat GPT, commit
92d63d4e8bb4df75c3b71618f31ddde2378b2bcd. 6 layers, 6 heads, 384 dims, 512-token context, 8,192-token BPE vocabulary; 26.3M parameters, float32. - Pretraining: 5,000 steps; validation 0.449 bits per byte.
- Story fine-tune: 1,409 steps on 100,000 request→story pairs; validation 0.415 bits per byte.
- Hardware: a single Apple Silicon Mac (MPS), about 2.4 hours end to end.
Training data
TinyStories (Eldan & Li, Microsoft Research, CDLA-Sharing-1.0): short stories written by GPT-3.5 and GPT-4 using only words a young child knows. Only the GPT-4 stories were used. The dataset is synthetic, so no real child's words are in it.
Before training, every story and its summary went through the same two filters that guard a child's screen in Chittu: a lexicon filter (crisis, personal information, unsafe content, secrets, identity, profanity) and an output filter. Anything either one flagged was removed. No thresholds, no exceptions:
| stories | share | |
|---|---|---|
| total | 2,494,822 | 100% |
| kept | 2,327,256 | 93.3% |
| removed: unsafe content | 61,645 | 2.5% |
| removed: identity | 51,765 | 2.1% |
| removed: output filter | 38,808 | 1.6% |
| removed: personal information | 7,328 | 0.3% |
| removed: crisis | 6,142 | 0.2% |
| removed: secrets | 1,868 | 0.1% |
| removed: profanity | 10 | <0.1% |
The filters are strict on purpose. They also removed harmless stories, such as a character saying "I love you, Mommy".
Evaluation
64 generations (completions from the base model, plus story requests written the way a child types), sampled at temperature 0.6, top-k 50, seed 42:
| check | result |
|---|---|
| lexicon filter hits | 0 of 64 |
| output filter failures | 0 of 64 |
These filters check for words and patterns. They don't judge whether a story is good, coherent or suitable for its age group, and no person has yet read and signed off the generations. General knowledge benchmarks were deliberately not run: a children's story model shouldn't have that knowledge.
Usage
The weights are safetensors and the tokenizer is a plain tiktoken ranks file, so
nothing here needs unpickling. You need nanochat's model code on the Python path (at or
near the commit above), plus torch, tiktoken and safetensors:
from load import load, story # load.py is in this repo
model, tok = load("path/to/chittu-story-v0-26m")
print(story(model, tok, "Write a story about a brave little sparrow."))
base, tok = load("path/to/chittu-story-v0-26m", "base") # the completion model
story() wraps the request in the chat format the model was fine-tuned on
(<|bos|><|user_start|>…<|user_end|><|assistant_start|>) and stops at <|assistant_end|>.
Sample output (temperature 0.6, seed 42):
Once upon a time, there was a brave little sparrow. He lived in a small village near a big hill. The brave sparrow loved to explore the world around him. One day, he decided to explore a new place…
Intended use
Research into small, offline, kid-locked story generators, and the creative/offline-stories feature of Chittu. Not a general-purpose assistant.
Licence
The model weights and load.py are released under Apache-2.0 (LICENSE). The training
data, TinyStories, is © Microsoft and licensed CDLA-Sharing-1.0, and is not redistributed
here. nanochat's code is MIT-licensed. "Chittu" and "Tulifo" are Tulifo names. Publishing
this model does not license them.