Token Garden Story 28M v1
A 27,795,968-parameter English story-completion model, trained from scratch as a small-model learning project. Give it a story opening, not a chat conversation. It has no instruction tuning, dialogue fine-tuning, or preference alignment. Basic continuations sometimes work; reliable storytelling and chatbot capability are not established.
Release: ev1lkow/token-garden-story-28m-v1, version v1.0.0, selected training run story-focus-v2.
Run locally
This repository uses its own readable PyTorch implementation, not a Transformers AutoModel integration. Download the versioned files together, then run the supplied script. No private training repository or account credentials are needed for inference.
python -m pip install huggingface_hub
hf download ev1lkow/token-garden-story-28m-v1 \
--revision v1.0.0 --local-dir token-garden-story-28m-v1
cd token-garden-story-28m-v1
python -m pip install -r requirements.txt
python story_generate.py --model-dir . \
--prompt "Once upon a time, a little fox found a key." \
--max-new-tokens 128 --temperature 0.8 --top-k 40 --seed 7 --device cpu
The download command uses the official Hub CLI. Use a virtual environment for installation. --device auto selects Apple MPS when available, otherwise CPU; --device mps explicitly requests Apple GPU inference. The supplied runtime does not expose CUDA.
From Python, in the downloaded directory:
from story_generate import load_model, generate
model, tokenizer, config = load_model(".", device="auto")
result = generate(
model, tokenizer,
"Once upon a time, a little fox found a key.",
max_new_tokens=128, temperature=0.8, top_k=40, seed=7,
)
print(result["continuation"])
Omitting the seed chooses a fresh seed and reports it in the result. Fixed seeds support repeatability on the same runtime/device, not guaranteed cross-device bitwise identity. The script generates locally from FP32 Safetensors weights and checks their configured hash and the tokenizer hash. Review custom code before executing it; hashes detect mismatched files, not an untrusted publisher. The default 128-token generation budget differs from the historical 200-token evaluation budget.
Architecture
| Property | Value |
|---|---|
| Parameters | 27,795,968 |
| Decoder layers / width | 8 / 512 |
| Attention | 8-head causal MHA, PyTorch SDPA |
| Context | 512 tokens |
| Position / normalization | RoPE / RMSNorm |
| Feed-forward | SwiGLU, hidden width 1,408 |
| Vocabulary | 4,096 byte-level BPE tokens |
| Embedding / output head | Tied weights |
The tokenizer was fitted only on 19,923 approved TinyStories TRAIN documents and then frozen. Generation retains the most recent 512 tokens and resets position indices when sliding; this is not long-context training or persistent memory.
Training data and selected lineage
The positive-weight next-token training source is TinyStories V2 GPT-4 TRAIN, revision f54c09fd23315a6f9c86f9dc80f725de7d8f9c64, file TinyStoriesV2-GPT4-train.txt. The upstream card identifies this version as GPT-4-generated. TinyStories deliberately targets simple stories and a vocabulary familiar to young children; it is not a corpus of sophisticated fiction. See Eldan and Li, 2023.
The local TinyStories inventory expanded from 517,925 to 1,513,937 and finally 2,706,374 TRAIN stories. Original validation stayed at 998 stories / 199,226 prediction targets. Data preparation included exact and approximate near-duplicate checks, train/validation separation, exact development-opening exclusions for expansions, and small complete-story AI spotchecks. These are not per-document human review, semantic decontamination proof, or a universal quality guarantee. Reserved test data was not used.
| Selected stage | Added updates | Physical target exposures | Nonzero-loss target exposures |
|---|---|---|---|
| modern-v1: random initialization | 14,000 | 179,383,130 | 179,383,130 |
| modern-v2: weights warm start | 14,000 | 177,367,094 | 177,367,094 |
| mixture study: tiny-only arm | 3,000 | 44,052,799 | 23,251,210 |
| focus-v1: physically Tiny-only | 12,000 | 152,528,390 | 152,528,390 |
| focus-v2: state continuation | 6,000 | 77,032,098 | 77,032,098 |
| Total selected lineage | 49,000 | 630,363,511 | 609,561,922 |
The 20,801,589-exposure difference is SimpleStories text physically processed with zero cross-entropy weight during the tiny-only mixture-study arm. The local mixed inventory also contains 1,802,574 SimpleStories TRAIN documents, but this is not positive-weight SimpleStories training of the released model. A separate SimpleStories validation subset was used diagnostically. Its pinned source is SimpleStories/SimpleStories at e63b8adc3b1a1bdc7cac5b500d150b71346b0628.
Updates from discarded mixed-data runs, other study arms, or earlier character models are not inherited. Sampling was with replacement: exposure totals are not distinct corpus coverage or learned knowledge. Training used PyTorch MPS BF16 autocast with FP32 master parameters and AdamW state; full validation remained FP32. The final 6,000 updates retained optimizer and sampling state and used a cosine learning-rate tail from 1e-5 to 3e-6. Training ended at its step budget, not a demonstrated ceiling. No new training was performed for this public export.
Evaluation: small, repeatedly used development sets
| Measure | Parent focus-v1 | Released focus-v2 |
|---|---|---|
| Fixed story development cases passing all criteria | 12 / 20 | 15 / 20 |
| Previously observed diagnostic cases passing all criteria | 15 / 20 | 13 / 20 |
| Full original validation loss, 199,226 targets | 1.084492 | 1.080184 |
| Full SimpleStories diagnostic loss, 501,676 targets | 2.670486 | 2.663877 |
Each story group uses 10 openings with seeds 7 and 19, temperature 0.8, top-k 40, and at most 200 new tokens. Passing requires all three: understandable, predominantly grammatical English; continuation of the opening's main character/situation; and at least two connected events without a central contradiction. Truncation or an unfinished ending alone is not failure. A pass means basic usability under these criteria, not high-quality prose.
These are actual output-bound AI readings, not human evaluations or independent benchmarks. Criteria disagreements received an additional reading and textual adjudication; identical outputs retained their prior judgments. The prompts were repeatedly observed during development, and reader disagreement was substantial. The diagnostic decline must be considered alongside the fixed-set gain. No statistical significance, stable chatbot performance, or 20/20 achievement is claimed. Checkpoint selection used full original validation loss, not these story scores.
Limitations and release contents
Expect repetitive wording, weak causality, character/object contradictions, abrupt endings, factual errors, and narrow vocabulary. Synthetic training data can carry biases. The model has no safety fine-tuning and is not certified appropriate for unsupervised use by children; a children's-story source is not a safety guarantee. It is intended for local story-completion experiments and learning, not factual or high-stakes assistance. Memorization has not been ruled out.
This is an inference release, not a complete training-reproduction package. It includes standalone runtime code, configuration, tokenizer, model-only Safetensors, compact training_metrics.json, verification.json, and release_manifest.json. Optimizer states, full training scripts/reports, raw corpora, and private project content are not included. See the verification file for the checks actually performed and their boundaries.
The released model results and original runtime code use Apache-2.0; see LICENSE. Source datasets retain their own terms, including TinyStories' CDLA-Sharing-1.0. See NOTICE for exact provenance and the distinction between model results and source data.