Llama-3.2-1B-MathCodeInstruct-5k
A Llama-3.2-1B fine-tune on 5k examples from MathLLMs/MathCodeInstruct, trained to solve math word problems with step-by-step natural-language reasoning interleaved with executable Python.
This is one of three sibling models trained on {5k, 10k, 20k}-example subsets of the same dataset, to study how fine-tuning data volume trades off against both math performance and general capability. See the training write-up for the full comparison across all three.
Training details
| Base model | unsloth/Llama-3.2-1B |
| Method | LoRA (r=16, α=16, dropout=0) on all attention + MLP projections, merged to full weights |
| Dataset | MathLLMs/MathCodeInstruct, 5k training examples |
| Epochs | 1 |
| Effective batch size | 16 (batch 1 × grad. accum. 16) |
| Learning rate | 2e-4, cosine schedule, warmup ratio 0.03 |
| Hardware | 1× RTX 4060 (8GB) |
| Framework | Unsloth + TRL SFTTrainer |
Benchmark results
All benchmarks run with lm-evaluation-harness, each at its standard published shot count, compared against the un-tuned base model.
| Benchmark | Llama-3.2-1B (base) | MathCodeInstruct-5k | Change |
|---|---|---|---|
| GSM8K | 5.8% | 7.4% | 🟢 +1.5% |
| ARC-Challenge | 36.9% | 36.8% | ⚪ -0.1% |
| HellaSwag | 64.2% | 63.8% | 🔴 -0.3% |
| WinoGrande | 60.8% | 62.4% | 🟢 +1.7% |
Speed: 40.67 tokens/sec (base model: 40.59 tokens/sec)
MMLU by category
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "OliverSundaram/Llama-3.2-1B-MathCodeInstruct-5k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")
messages = [
{"role": "system", "content": "Below is a math problem. Please solve it step by step."},
{"role": "user", "content": "If a train travels 60 miles in 45 minutes, what is its speed in miles per hour?"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
output = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Limitations
- Trained on a single epoch of a 5k-example subset — not intended to be a general-purpose assistant.
- MMLU/ARC/HellaSwag/WinoGrande scores reflect a small 1B-parameter base model and should be read relative to the base model's own scores, not against much larger models.
- No safety alignment or RLHF was applied beyond what the base Llama-3.2-1B already has.