MiniCPM5-2B — GGUF quants

Full llama.cpp quant sweep of openbmb/MiniCPM5-2B, OpenBMB's dense 2B model for on-device and local deployment. 30 files — the BF16 source plus 29 quants from Q8_0 down to Q1_0 — all cut from the official BF16 release, with the importance matrix computed on the BF16 weights.

The source model uses the standard LlamaForCausalLM architecture — no custom kernels and no model-code fork are needed; these files load directly in llama.cpp, Ollama and LM Studio. (OpenBMB also publishes their own MiniCPM5-2B-GGUF; this repo complements it with the complete quant sweep including the IQ and extreme quants. For vLLM, SGLang or transformers, use the source repo directly.)

Which file should I get?

All 30 files combined: 40.15 GB.

Shortcuts: default Q4_K_M (1.45 GB) · lossless Q8_0 (2.50 GB) · smallest usable IQ2_M (0.90 GB) · 3-bit sweet spot IQ3_M (1.14 GB).

Usage

Run straight from the Hub:

llama-cli -hf NANI-Nithin/MiniCPM5-2B-GGUF:Q4_K_M -p "Hello"

Serve an OpenAI-compatible endpoint:

llama-server -hf NANI-Nithin/MiniCPM5-2B-GGUF:Q4_K_M

Or download a file first:

huggingface-cli download NANI-Nithin/MiniCPM5-2B-GGUF MiniCPM5-2B-Q4_K_M.gguf --local-dir .
llama-cli -m MiniCPM5-2B-Q4_K_M.gguf -p "Hello"

These are llama.cpp GGUF files. They are not loaded by vLLM, SGLang or transformers — for those backends use the source repo.

About MiniCPM5-2B

MiniCPM5-2B is the second model in the MiniCPM5 series from OpenBMB, following MiniCPM5-1B. It is a dense 2B Transformer built for local assistants, coding agents, tool-use workflows and reasoning scenarios where a compact footprint matters. Per the source model card:

• 2,516,756,480 parameters (1,981,982,720 non-embedding), 42 layers, GQA with 16 query heads and 2 KV heads
• 131,072-token context with native long-context support
• English and Chinese
• Trained with the UltraData tiered data management recipe (arXiv:2602.09003): base + mid-training on the released web datasets, then 400B tokens of deep-thinking SFT, RL, and On-Policy Distillation (OPD) of 16 RL teacher models into the release checkpoint
• OpenBMB's own evaluation reports 2B-class open-source SOTA with an average score of 53.9 in their comparison set, competitive with 4B-class models, with particular strengths in code reasoning, math, long-context understanding, tool use and agentic tasks

Quantization notes

• Quantized from the official BF16 release on 2026-09-07
• Importance matrix computed on the BF16 weights, so the IQ and sub-Q4 quants benefit from calibration
• All 29 quant types of the standard sweep are present; nothing was skipped for size

Citation

If you use this model, please cite the source work:

@article{minicpm4,
  title={Minicpm4: Ultra-efficient llms on end devices},
  author={MiniCPM, Team},
  journal={arXiv preprint arXiv:2506.07900},
  year={2025}
}

Disclaimer

Per the source model card, outputs are generated text and may be inaccurate, biased, or offensive; responses on sensitive topics are not professional advice. The model is provided AS IS, without warranty of any kind; users must use it only for lawful, compliant and ethical purposes.

License

Apache-2.0, matching the source model.