RonMicro-LLM-Story (Phase 1)
A small GPT-2 style language model trained on TinyStories dataset for generating children's stories.
Model Details
- Model Type: GPT-2 Causal Language Model
- Parameters: ~40M
- Training Data: TinyStories (5% subset, ~105K stories)
- Vocabulary Size: 25,913 tokens
- Context Length: 512 tokens
- Training Epochs: 3
- Language: English
Training Details
- Framework: Transformers (Hugging Face)
- Tokenizer: Custom BPE trained on TinyStories
- Architecture:
- 6 transformer layers
- 384 embedding dimensions
- 6 attention heads
- 1536 FFN dimensions
Usage
from transformers import pipeline
# Load the model
generator = pipeline("text-generation", model="endurasolution/ronmicro-llm-story")
# Generate a story
story = generator(
"Once upon a time",
max_new_tokens=150,
temperature=0.7,
repetition_penalty=1.3,
no_repeat_ngram_size=3,
do_sample=True
)
print(story[0]["generated_text"])
Example Outputs
Prompt: "Once upon a time" Output: "Once upon a time, there was a little boy named Timmy. He loved to play with his toy cars and trucks all day long..."
Limitations
- Trained on only 5% of TinyStories (Phase 1)
- May generate repetitive text occasionally
- Best for short children's stories (100-200 words)
- Limited to simple vocabulary and grammar
Next Steps
Phase 2 training in progress with 20% data and 5 epochs for improved quality.
Citation
Built using TinyStories dataset:
@article{eldan2023tinystories,
title={TinyStories: How Small Can Language Models Be and Still Speak Coherent English?},
author={Eldan, Ronen and Li, Yuanzhi},
journal={arXiv preprint arXiv:2305.07759},
year={2023}
}
License
MIT License