OpenHermes-Llama-3.2-1B
Model Description
Thanks Redmond.AI for GPU Sponsor!
This model is a fine-tuned version of meta-llama/Llama-3.2-1B on the OpenHermes-2.5 dataset. It is based on the Llama 3.2 architecture, which is an optimized transformer model designed for multilingual dialogue use cases, including agentic retrieval and summarization tasks.
Key Features:
- Base Model: Meta Llama 3.2 1B
- Fine-tuning Dataset: OpenHermes 2.5
- Architecture: Auto-regressive language model with optimized transformer architecture
- Params: 1.23B
- Context Length: 128k
- Input/Output Modalities: Multilingual Text and code
- Supported Languages: Primarily English, with potential for other languages supported by Llama 3.2
Intended Uses & Limitations
This model is intended for commercial and research use in multiple languages, particularly suited for assistant-like chat and agentic applications such as knowledge retrieval, summarization, and query rewriting. It inherits the capabilities and limitations of both the Llama 3.2 1B base model and the OpenHermes 2.5 dataset.
Out of Scope:
- Use that violates applicable laws or regulations
- Use prohibited by the Acceptable Use Policy and Llama 3.2 Community License
- Use in unsupported languages without proper evaluation and safety measures
Training Procedure
Training Data
The model was fine-tuned on the OpenHermes 2.5 dataset, which contains 1M primarily synthetically generated instruction and chat samples. This dataset is a compilation of various open-source datasets and custom-created synthetic datasets, designed to enhance the model's performance in instruction-following and chat scenarios.
Training Results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.1101 | 0.0003 | 1 | 0.9499 |
| 0.7977 | 0.5000 | 1438 | 0.8729 |
| 0.8338 | 1.0000 | 2876 | 0.8647 |
| 0.7714 | 1.4981 | 4314 | 0.8637 |
| 0.8305 | 1.9983 | 5752 | 0.8612 |
| 0.6801 | 2.4963 | 7190 | 0.8631 |
Evaluation Results
The model achieves a final validation loss of 0.8631.
Ethical Considerations and Limitations
Users should be aware of potential biases in the training data and exercise caution when deploying the model, especially in sensitive applications. The model's outputs should be carefully monitored and filtered for inappropriate content.
For more information on the base Llama 3.2 model, please refer to the official Llama 3.2 model card.
🌐 Website
You can find more of my models, projects, and information on my official website:
🚀 Prompt Hub
Need high-quality prompts for image models and LLMs? Explore findgoodprompt.com.
💖 Support My Work
If you find this model useful, please consider supporting my work. It helps me cover server costs and dedicate more time to new open-source projects.
- Patreon: Support on Patreon
- Ko-fi: Buy me a Ko-fi
- Buy Me a Coffee: Buy me a Coffee
Explore More
Discover more of my models, projects, and practical AI resources:
- AI tools directory: JustGoodAI — a curated directory of useful AI tools.