Tiny-Cowboy-1.1b-v0.1
Tiny-Cowboy-1.1b-v0.1 is a specialized language model designed for generating cowboy-themed content. Developed by phanerozoic, this model is fine-tuned from TinyLlamaTinyLlama-1.1B-Chat-v1.0, optimized for environments with limited computing resources.
Performance
The model excels in generating engaging cowboy narratives and demonstrates a strong grasp of cowboy culture and lifestyle. However, it is less effective in general language tasks, especially in scientific and technical domains.
Direct Use
Ideal for thematic language generation, particularly in applications where cowboy culture and storytelling are central. Less suited for general-purpose use or scenarios requiring detailed, accurate scientific explanations.
Context Setting and Interaction Guidelines
Tiny-Cowboy-1.1b-v0.1, being a narrowly focused and somewhat limited-performance model, benefits from an initial context-setting message. This setup involves a predefined assistant message that establishes its cowboy identity at the start of each interaction. This strategy is crucial for priming the model to maintain its cowboy theme throughout the conversation. It's important to note that the model has been fine-tuned for a cowboy style of speaking, so explicit instructions on how to respond in a cowboy manner are unnecessary.
Initial Context Setting:
-
text: | Assistant: Howdy! I'm your cowboy assistant, ready to talk all things Wild West. What cowboy queries can I lasso for you today? example_title: "Initiating Cowboy Themed Conversation"
-
text: | Assistant: Yeehaw! Let's dive into the cowboy world. Ask me anything about cowboys, ranches, or the Wild West! example_title: "Engaging in Cowboy Themed Dialogue"
The introduction by the assistant sets the thematic tone, guiding the user to interact within the cowboy context.
Training Data
Incorporates a dataset focused on cowboy and Wild West themes, derived from the foundational TinyLlama-1.1B model.
Custom Stopping Strings
Custom stopping strings were used to refine output quality:
- "},"
- "User:"
- "You:"
- "\nUser"
- "\nUser:"
- "me:"
- "user"
- "\n"
Training Hyperparameters and Fine-Tuning Details
- Base Model Name: TinyLlamaTinyLlama-1.1B-Chat-v1.0
- Base Model Class: LlamaForCausalLM
- Projections: gate, down, up, q, k, v, o
- LoRA Rank: 16
- LoRA Alpha: 32
- True Batch Size: 4
- Gradient Accumulation Steps: 1
- Epochs: 1
- Learning Rate: 3e-4
- LR Scheduler: Linear
- LLaMA Target Projections: All targets modified
- Loss: 2.096
- Stop Step: 42
Limitations
While adept at cowboy-themed content, Tiny-Cowboy-v0.1 struggles with topics outside its specialty, particularly in scientific and technical areas. The model tends to incorporate cowboy elements into responses, regardless of the question's relevance.
Compute Infrastructure
Efficiently trained, demonstrating the feasibility of specialized model training in resource-constrained environments.
Results
Successfully generates cowboy-themed responses, maintaining thematic consistency. However, it shows limitations in handling more complex, non-cowboy-related queries.
Summary
Tiny-Cowboy-1.1b-v0.1 is a significant development in thematic, lightweight language models, ideal for cowboy-themed storytelling and educational purposes. Its specialization, however, limits its applicability in broader contexts, particularly where accurate, technical knowledge is required.
Acknowledgments
Special thanks to the TinyLlama-1.1B team, whose foundational work was instrumental in the development of Tiny-Cowboy-v0.1.