TinyViking-1.1B-v0.1
TinyViking-1.1B-v0.1 is a specialized language model designed for generating Viking-themed content. Developed by phanerozoic, this model is fine-tuned from TinyLlamaTinyLlama-1.1B-Chat-v1.0, optimized for environments with limited computing resources.
Performance
TinyViking is capable of generating engaging Viking narratives, reflecting an understanding of Viking culture. However, it is not designed for general language tasks and may struggle with complex scientific or technical queries.
Direct Use
Ideal for thematic language generation, particularly in settings like NPCs in games, where fun and thematic engagement are prioritized over detailed factual accuracy.
Training Data
Trained on "The Saga of Grettir the Strong: Grettir's Saga" to ensure authentic thematic content.
Custom Stopping Strings
Custom stopping strings are employed to enhance output quality:
- "},"
- "User:"
- "You:"
- "\nUser"
- "\nUser:"
- "me:"
- "user"
- "\n"
Training Hyperparameters and Fine-Tuning Details
- Learning Rate: 2e-5
- Epochs: 1
- Training Duration: Approximately 5.6 minutes on an RTX 6000 Ada GPU
- LoRA Rank: 2048
- LoRA Alpha: 4096
- LoRA Dropout: 0.05
- Cutoff Length: 256
- Batch Size: 4 (micro batch size)
- Warmup Steps: 8
- Optimizer: adamw_torch
- Gradient Accumulation Steps: 1
Limitations
Specialized in Viking dialect and narratives, TinyViking is less effective outside its thematic focus.
Compute Infrastructure
Trained on an RTX 6000 Ada Lovelace GPU
Results
Successfully generates Viking-themed responses, maintaining thematic consistency while displaying improved coherence and depth over previous models due to advancements in dataset generation and parsing.
Summary
TinyViking-1.1B-v0.1 shows an improvement in quality compared to earlier thematic models, thanks to a new dataset generation method that helps to conserve the base model's already tenuous ability to hold a conversation. While it excels in Viking-themed interactions, its specialized focus limits broader application.
Acknowledgments
Gratitude to the TinyLlama team, whose foundational work was, as always, essential for developing TinyViking.