Pige-SLM 8.9M TinyStories v0.1
Pige-SLM 8.9M TinyStories v0.1 is a small decoder-only Transformer language model trained from scratch for text generation.
The model was built and trained from scratch using PyTorch on a consumer GPU.
Model specifications
| Specification | Value |
|---|---|
| Parameters | 8,900,608 |
| Architecture | Decoder-only Transformer |
| Layers | 6 |
| Attention heads | 8 |
| Embedding dimension | 256 |
| Context length | 256 tokens |
| Vocabulary size | 8,000 |
| Training tokens | ~53.2M |
| Training examples | 250,000 |
| Best validation loss | 2.1876 |
Training
The model was trained on the TinyStories dataset.
The tokenizer uses an 8,000-token vocabulary and the model was trained using next-token prediction.
The best validation loss was achieved at training step 11,500.
Intended use
This model is intended for:
- experimentation with small language models
- educational purposes
- text-generation experiments
- learning about Transformer architectures
- research and development with small language models
Limitations
This is a very small language model and should not be expected to perform like modern large language models.
It was trained primarily on children's stories, so its outputs tend to resemble the style and structure of the training data.
The model is not instruction-tuned and is not designed to function as a general-purpose conversational assistant.
It may produce:
- repetitive text
- incorrect information
- incoherent passages
- grammatical errors
- abrupt changes in topic or characters
Example
Prompt:
Once upon a time