Gemma-4-E4B-Claude-4.6-Opus-Reasoning-Distilled
Fine-tune of Gemma 4 E4B trained on Claude 4.6 Opus reasoning traces. The goal: take a compact 4B model and teach it to actually think before answering.
💡 What this is
Standard Gemma 4 E4B is already solid. This fine-tune pushes it toward a more deliberate, structured reasoning style by training on ~2.3k high-quality Chain-of-Thought samples distilled from Claude 4.6 Opus.
The model learns to plan inside <think> tags before committing to a final
answer — fewer impulsive responses, more structured breakdowns.
<think>
1. What is actually being asked here?
2. What are the constraints and edge cases?
3. Step-by-step plan...
4. Verify the logic holds.
</think>
Final answer here.
🗺️ Pipeline
google/gemma-4-E4B-it
│
▼
SFT + QLoRA 4-bit (Unsloth)
│ loss masked to responses only
▼
Gemma-4-E4B-Claude-4.6-Opus-Reasoning-Distilled
│
▼
exported as GGUF (Q4_K_M + Q8_0)
⚙️ Training Details
| Parameter | Value |
|---|---|
| Base model | google/gemma-4-E4B-it |
| Framework | Unsloth |
| Method | SFT + QLoRA (4-bit) |
| Dataset | nohurry/Opus-4.6-Reasoning-3000x-filtered |
| Hardware | RTX 5060 Ti 16GB |
| LoRA rank / alpha | 16 / 16 |
| Epochs | 3 |
| Max seq length | 2048 |
| Optimizer | adamw_8bit |
| Learning rate | 2e-4 |
| LR scheduler | cosine |
| Loss masking | train_on_responses_only |
📚 Dataset
| Dataset | Description |
|---|---|
| nohurry/Opus-4.6-Reasoning-3000x-filtered | ~2.3k filtered Claude 4.6 Opus reasoning trajectories covering math, logic, and coding |
🚀 Run it
Ollama:
ollama run hf.co/arsovskidev/Gemma-4-E4B-Claude-4.6-Opus-Reasoning-Distilled
llama.cpp:
./llama-cli -hf arsovskidev/Gemma-4-E4B-Claude-4.6-Opus-Reasoning-Distilled \
--temp 1.0 --top-p 0.95 --top-k 64
✅ Good at
- Multi-step math and logic problems
- Code problem decomposition and debugging
- Tasks where showing reasoning is more valuable than raw speed
- Structured analysis of complex prompts
⚠️ Limitations
- Text only — multimodal capabilities of the base model are not trained here
- Small dataset — treat this as a focused reasoning fine-tune, not a general-purpose upgrade
- Still an LLM — hallucinations happen, especially on factual recall outside the training domain
📜 License
Apache 2.0 + Gemma Terms of Use.
"Claude" is a trademark of Anthropic. This project is not affiliated with or endorsed by Anthropic — the name refers to the reasoning distillation data source only.
🙏 Acknowledgements
Unsloth for making this feasible on consumer hardware, and nohurry for the dataset.
📖 Citation
@misc{arsovskidev_gemma4_opus_distilled,
title = {Gemma-4-E4B-Claude-4.6-Opus-Reasoning-Distilled},
author = {arsovskidev},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/arsovskidev/Gemma-4-E4B-Claude-4.6-Opus-Reasoning-Distilled}}
}