BoostedV1
Continued LoRA training of deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B for improved reasoning and code generation. This model is the merged output of the boostedv1train pipeline: 400 MLX LoRA steps (Apple Silicon) + 550 Phase-1 continuation steps (dual RTX 3090, bf16).
Architecture
- Base: DeepSeek-R1-Distill-Qwen-1.5B (Qwen2ForCausalLM, 1.5B params)
- LoRA rank 8, alpha 160, on layers 20-27 (q/k/v/o + gate/up/down)
- Merged into a standalone model (no LoRA needed at inference)
Training
| Stage | Platform | Steps | Batch | Seq len | Data |
|---|---|---|---|---|---|
| MLX run 1 | Apple Silicon | 150 | 4 | 512 | OpenCodeInstruct |
| MLX run 2 | Apple Silicon | 250 | 2 | 1024 | OpenCodeInstruct |
| Phase 1 | 2x RTX 3090 | 550 | 16 (eff) | 2048 | OpenThoughts + OpenR1-Math + OpenCodeInstruct |
Evaluation
| Benchmark | Score |
|---|---|
| GSM8K | 46.0% |
| HumanEval (pass@1) | 7.3% |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('auryn-macmillan/boostedv1') tok = AutoTokenizer.from_pretrained('auryn-macmillan/boostedv1') inputs = tok('What is 2+2?', return_tensors='pt') out = model.generate(**inputs, max_new_tokens=128) print(tok.decode(out[0]))
Notes
- Standard Qwen2 architecture, no custom code, no trust_remote_code needed.
- Trained in an isolated container; repo contains no training code or data.