MiniCPM5-2B Distilled Reasoning (GGUF)
Quantized GGUF version of jigs97022/minicpm5-2b-distilled-reasoning. Optimized for local CPU/GPU inference with llama.cpp, Ollama, LM Studio, and vLLM.
Model Origin
This model was distilled from three frontier models (Qwen3.8-Max, GLM-5.2, Kimi K3) onto MiniCPM5-2B using 10K high-quality reasoning traces. It excels at structured chain-of-thought problem solving in math, code, logic puzzles, and tool-use scenarios.
Quick Start
Ollama (Recommended)
ollama run hf.co/jigs97022/minicpm5-2b-distilled-reasoning-gguf
LM Studio
- Search "jigs97022/minicpm5-2b-distilled-reasoning-gguf" in the search bar
- Download the Q4_K_M quantization (or Q6_K / Q8_0 for higher quality)
- Load and chat
llama.cpp / Python
from llama_cpp import Llama
llm = Llama.from_pretrained( repo_id="jigs97022/minicpm5-2b-distilled-reasoning-gguf", filename="*q4_k_m.gguf", n_ctx=4096, n_gpu_layers=-1 # Set to 0 for CPU-only )
output = llm.create_chat_completion( messages=[{"role": "user", "content": "If 3x + 7 = 22, what is x? Show step-by-step reasoning."}], max_tokens=512, temperature=0.7 ) print(output["choices"][0]["message"]["content"])
vLLM
vllm serve jigs97022/minicpm5-2b-distilled-reasoning-gguf
--max-model-len 4096
--gpu-memory-utilization 0.8
Quantization Details
| File | Size | Quality | Best For |
|---|---|---|---|
| Q4_K_M | 1.56 GB | Recommended default | Laptop CPU/GPU, balanced speed/quality |
| Q6_K | 2.07 GB | High quality | Desktops with 4GB+ VRAM, best balance |
| Q8_0 | 2.68 GB | Near-lossless | Benchmarking, when quality matters most |
Hardware Requirements
| Quant | RAM/VRAM Min | Inference Speed (T4) |
|---|---|---|
| Q4_K_M | 2 GB | ~40 tok/s |
| Q6_K | 3 GB | ~35 tok/s |
| Q8_0 | 3.5 GB | ~30 tok/s |
Chat Template
Uses native MiniCPM5 formatting. Do NOT override the chat template in llama.cpp/Ollama; the model expects its built-in template.
Limitations
- Context window limited to 4096 tokens
- Trained on 10K samples; may underperform full-corpus variant on rare edge cases
- Knowledge cutoff inherited from base MiniCPM5 (~2024)
- May produce verbose think sections; adjust temperature down to 0.5 if you prefer concise outputs
License
Apache-2.0 (inherited from base MiniCPM5-2B). Verify upstream distillation dataset license before commercial deployment.
Acknowledgments
- OpenBMB for MiniCPM5-2B
- r0b0tlab for the multi-teacher distillation dataset
- Unsloth for accelerated training and GGUF export
- llama.cpp team for GGUF format specification