Model Overview
LlamaTron RS1 Nemesis is a medical reasoning model produced by fine-tuning meta-llama/Llama-3.2-1B-Instruct on the Medical-Reasoning-SFT-MiniMax-M2.1 dataset using QLoRA. The dataset contains 204,773 clinical reasoning conversations with full chain-of-thought traces covering differential diagnosis, treatment planning, pharmacology, and clinical case analysis.
Despite being a 1 billion parameter model, it handles complex clinical questions with structured and coherent reasoning.
Training Results
| Step | Train Loss | Validation Loss |
|---|---|---|
| 500 | 1.5759 | 1.6126 |
| 1000 | 1.5176 | 1.5538 |
| 1500 | 1.4805 | 1.5256 |
| 2000 | 1.4795 | 1.5060 |
| 2500 | 1.4508 | 1.4939 |
| 3000 | 1.4534 | 1.4815 |
| 3500 | 1.4384 | 1.4739 |
| 4000 | 1.4228 | 1.4663 |
| 4500 | 1.4251 | 1.4605 |
| 5000 | 1.4301 | 1.4567 |
| 5500 | 1.4102 | 1.4545 |
| 6000 | 1.4246 | 1.4538 |
| 6271 | 1.4200 | 1.4500 |
Loss decreased consistently across all steps with train and validation loss tracking closely. No overfitting observed.
Dataset
Trained on Medical-Reasoning-SFT-MiniMax-M2.1 released by Maziyar Panahi under the OpenMed initiative.
| Property | Value |
|---|---|
| Total Samples | 204,773 |
| Estimated Tokens | ~621 Million |
| Format | Multi-turn chat with chain-of-thought reasoning |
| License | Apache 2.0 |
| Topics | Differential diagnosis, treatment planning, pharmacology, clinical case analysis |
How to Use
Load the Model
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model_id = "Rumiii/LlamaTron_RS1_Nemesis_1B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
messages = [
{
"role": "system",
"content": "You are LlamaTron RS1 Nemesis, a knowledgeable and compassionate medical AI assistant. Provide accurate, evidence-based medical information clearly and helpfully."
},
{
"role": "user",
"content": "What are the early symptoms of Type 2 Diabetes?"
},
]
output = pipe(
messages,
max_new_tokens=400,
do_sample=True,
temperature=0.7,
top_p=0.9,
)
print(output[0]["generated_text"][-1]["content"])
Repository
The full training code, merging scripts, and inference interface are available on GitHub: github.com/sufirumii/LlamaTron-RS1-Nemesis-1B
Limitations
- This model is intended for research and educational purposes only
- It is not a substitute for professional medical advice, diagnosis, or treatment
- The model was trained with a maximum sequence length of 512 tokens which may limit performance on longer clinical texts
- Always consult a qualified healthcare provider for medical decisions
Credits
- Dataset: Maziyar Panahi and the OpenMed initiative for releasing the Medical-Reasoning-SFT-MiniMax-M2.1 dataset under Apache 2.0
- Base Model: Meta AI for releasing Llama-3.2-1B-Instruct
- Libraries: Hugging Face Transformers, PEFT, TRL, BitsAndBytes, Accelerate
License
Apache 2.0 — see LICENSE for details.