Model Details
- Model Developer: Aayan Mishra
- Model Type: Causal Language Model
- Architecture: Transformer with Rotary Position Embeddings (RoPE), SwiGLU activation, RMSNorm, and Attention QKV bias
- Parameters: 7 billion total
- Layers: 32
- Attention Heads: 24 for query and 4 for key-value (Grouped Query Attention)
- Vocabulary Size: Approximately 151,646 tokens
- Context Length: Supports up to 128,000 tokens
- Languages Supported: Primarily English, with capabilities in other languages
- License: MIT
Training Details
Athena-R3-7B was fine-tuned using the Unsloth framework on a single NVIDIA A100 GPU. The fine-tuning process involved 60 epochs over approximately 90 minutes, utilizing a curated dataset focused on reasoning tasks, including mathematical problem-solving and logical inference. This approach aimed to bolster the model's proficiency in complex reasoning and analytical tasks.
Intended Use
Athena-R3-7B is designed for a variety of applications, including but not limited to:
- Advanced Reasoning: Assisting with complex problem-solving and logical analysis.
- Academic Support: Providing explanations and solutions for mathematical and scientific queries.
- General NLP Tasks: Engaging in text completion, summarization, and question-answering tasks.
- Data Interpretation: Offering insights and explanations for data-centric inquiries.
While Athena-R3-7B is a powerful tool for various applications, it is not intended for real-time, safety-critical systems or for processing sensitive personal information.
How to Use
To utilize Athena-R3-7B, ensure that you have the latest version of the transformers library installed:
pip install transformers
Here's an example of how to load the Athena-R3-7B model and generate a response:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "Spestly/Athena-R3-7B"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
prompt = "Explain the concept of entropy in thermodynamics."
messages = [
{"role": "system", "content": "You are Athena, an AI assistant designed to be helpful."},
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
**model_inputs,
max_new_tokens=512
)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
Limitations
Users should be aware of the following limitations:
- Biases: Athena-R3-7B may exhibit biases present in its training data. Users should critically assess outputs, especially in sensitive contexts.
- Knowledge Cutoff: The model's knowledge is current up to August 2024. It may not be aware of events or developments occurring after this date.
- Language Support: While the model supports multiple languages, performance is strongest in English.
Acknowledgements
Athena-R3-7B builds upon the work of the DeepSeek team, particularly the DeepSeek-R1-Distill-Qwen-7B model. Gratitude is also extended to the open-source AI community for their contributions to tools and frameworks that facilitated the development of Athena-R3-7B.
License
Athena-R3-7B is released under the MIT License, permitting wide usage with proper attribution.
Contact
- Email: athena@aayanmishra.com