lilstory
a tiny 8m parameter llama-style model trained from scratch on tinystories. it writes short simple stories that mostly make sense and then kinda wander off.
run it
ollama run navthings/lilstory
or use lilstory.gguf with llama.cpp. the transformers version (model.safetensors + config + tokenizer) is in here too.
or try it in your browser: https://navthings.github.io/playground/
details
4 layers, 256 wide, 4 attention heads, 8000 token bpe tokenizer i trained myself. trained on a macbook with adamw, lr 1e-3, batch 32.
used in my paper: the effect of corpus size on the performance of a llm