Tiny Knight-1.1b-v0.1
Tiny Knight-1.1b-v0.1 is a specialized language model crafted for generating knight and medieval-themed content. This iteration is built upon the foundations of TinyLlama-1.1B-Chat-v1.0, tailored to operate within environments constrained by computing resources.
Performance
While this model excels in creating knight-themed narratives, its specialization, however, limits its effectiveness in broader language tasks, especially those requiring detailed knowledge outside the medieval theme.
Direct Use
Tiny Knight-1.1b-v0.1 is particularly suited for generating content within medieval, knightly, or fantasy settings, ideal for storytelling, educational content, and thematic exploration. It is not recommended for general-purpose tasks or technical domains.
Context Setting and Interaction Guidelines
Given its specialized nature, Tiny Knight-1.1b-v0.1 benefits significantly from detailed context-setting. Providing a rich thematic backdrop in prompts enhances the model's performance, guiding it to generate more accurate and immersive content.
Training Data
Incorporates a dataset focused on knightly tales, medieval history, and literature, derived from the foundational TinyLlama-1.1B model.
Custom Stopping Strings
Custom stopping strings were used to refine output quality:
- "},"
- "User:"
- "You:"
- "\nUser"
- "\nUser:"
- "me:"
- "user"
- "\n"
Training Hyperparameters and Fine-Tuning Details
- Base Model Name: TinyLlama-1.1B-Chat-v1.0
- Base Model Class: LlamaForCausalLM
- Projections: gate, down, up, q, k, v, o
- LoRA Rank: 16
- LoRA Alpha: 32
- True Batch Size: 32
- Gradient Accumulation Steps: 1
- Epochs: 0.18
- Learning Rate: 3e-4
- LR Scheduler: Linear
- Step: 75
- Loss: 1.87
Limitations
While adept at producing themed content, Tiny Knight-1.1b-v0.1's applicability is limited outside its specialized domain of knights and medieval themes.
Summary
Tiny Knight-1.1b-v0.1 represents a significant advancement in thematic language models, offering a specialized tool for exploring the medieval era. Its emphasis on context for optimal performance and the use of custom stopping strings make it a sophisticated asset for generating historically rich content.
Acknowledgments
Special thanks to the TinyLlama-1.1B team, whose pioneering work laid the groundwork for the creation of Tiny Knight-1.1b-v0.1.