Poetry Phisch
A 3M-parameter language model built with Phisch, trained on English poetry from Project Gutenberg.
Built with Phisch, a small open-source LLM written in Rust with the Burn framework. This model serves as a simple demonstration of the obtainable outputs of a full training pipeline with Phisch.
Try it
You'll need to compile phisch from our GitHub and the three files from this repository: model.safetensors, config.json, tokenizer.json.
Compiling it requires Rust and Cargo on your system. From the cloned repository, cargo install --path . installs the phisch command, the README has more details.
Put the three files in one directory and point phisch at it:
phisch generate "The bird" -a /path/to/poetry-phisch --checkpoint model.safetensors
Generation defaults to top-k 40 and temperature 0.8. You can also serve the model over HTTP with phisch serve, see USAGE.md.
Model details
| Architecture | Llama-style decoder (Phisch): RoPE, pre-norm RMSNorm, SwiGLU, tied embeddings, no biases |
| Parameters | 3,147,008 |
| Layers | 2 |
| Hidden Size | 256 |
| Feed-forward Size | 1024 |
| Attention Heads | 4 |
| Context length | 256 |
| Tokenizer | Byte-level BPE, 4096 tokens |
| Precision | f32 (safetensors export) |
Training
| Corpus | ~83 million tokens (train split), 4.4 million validation |
| Tokens seen | ~500 million (2 runs × 3 epochs) |
| Validation loss | 3.6929 |
| Optimizer | AdamW, batch 8 × 256 tokens, gradient clipping 1.0, dropout 0.1 |
| Schedule | Run 1: lr 1e-3 → 5e-5 cosine. Run 2, from run 1's best checkpoint: lr 3e-4 → 2e-5 cosine. 200 warmup steps each |
| Training hardware | NVIDIA GeForce GTX 1660 6GB, ~4.5 hours over both runs |
Data
2,230 English poetry books from Project Gutenberg, chosen by subject and bookshelf. A book was only included if it is public domain in the USA and every credited contributor (author, translator, editor) died 70 or more years ago (1955 or earlier). Books with an unknown death year were left out. Gutenberg headers, license text and transcriber notes were stripped.
Limitations
- English only.
- It continues text, it's not a chat or instruction model.
- It only sees the last 256 tokens.
- At this size, expect the form and rhythm of poetry more than coherent meaning.
- The training text is old, so the model reproduces the language and views of its period.
License
The Phisch code is licensed under Apache-2.0.