MiniCPM5-2B — NVFP4 W4A16 (GPTQ)
NVFP4 weight-only (W4A16) quantization of openbmb/MiniCPM5-2B
— 4-bit float weights in 16-element blocks with FP8 block scales, activations
in bf16, lm_head + embeddings kept in bf16. Rounding is GPTQ (Hessian-aware),
not RTN, which recovers most of the quality NVFP4 otherwise loses on a small
model. Produced with llm-compressor 0.13.
- 2.03 GiB on disk (bf16 base is 4.68 GiB — 57 % smaller)
- Serves on vLLM (
compressed-tensors); on SM120 the weight-only path currently decodes via the Marlin kernel to a bf16 GEMM (no native FP4 compute), so the benefit here is footprint, not raw speed - Coding cost vs bf16: ~2.4 pp HumanEval / ~4.8 pp MBPP — see the eval
Pick this if you need to fit near 2 GB (e.g. an 8 GB card with a large KV cache). If you can spare ~0.8 GB, the FP8 build is near-lossless.
Evaluation
lm-evaluation-harness, vLLM 0.26 backend, greedy, median of 3 draws with the
range (the harness is non-deterministic run-to-run even at greedy). All arms
measured in one session against the released checkpoint. HumanEval-instruct
pass@1/create_test (n = 164); MBPP base 3-shot (n = 500).
| build | HumanEval-inst | MBPP (3-shot) | size | Δ HE / MBPP |
|---|---|---|---|---|
| bf16 base | 86.59 % (85.98–86.59) | 50.60 % (50.40–51.00) | 4.68 GiB | — |
| FP8-dynamic | 84.76 % (84.15–85.37) | 48.80 % (48.80–49.00) | 2.84 GiB | −1.8 / −1.8 pp |
| NVFP4-W4A16 GPTQ (this) | 84.15 % (81.71–84.15) | 45.80 % (45.60–46.40) | 2.03 GiB | −2.4 / −4.8 pp |
| NVFP4-W4A16 RTN | 79.88 % (77.44–79.88) | 41.20 % (41.20–41.80) | 2.03 GiB | −6.7 / −9.4 pp |
| mixed (MLP-NVFP4 + attn-FP8) | 84.15 % (82.93–85.98) | 46.60 % (46.00–46.80) | 2.20 GiB | −2.4 / −4.0 pp |
GPTQ rounding recovers ~4.3 pp (HumanEval) / ~4.6 pp (MBPP) over plain RTN. A 2.5 B dense model still can't fully absorb 4-bit weights — the residual gap, especially on MBPP, is the cost of the extra ~0.8 GB saved vs FP8. A mixed-precision build (4-bit MLP + 8-bit attention) was tried and gives no advantage — same HumanEval, 0.17 GiB larger — so it is not published.
Full comparison, serving notes, and a fine-tune experiment that regressed the base's coding (so these quants target the released checkpoint): https://github.com/t-timms/minicpm5-2b-quants
Usage
vllm serve Ttimms/MiniCPM5-2B-NVFP4 --max-model-len 32768 --kv-cache-dtype fp8
On WSL, set VLLM_USE_V2_MODEL_RUNNER=0 (the V2 runner needs UVA, which WSL
disables).
Method & provenance
- Quantizer:
llm-compressor0.13,GPTQModifier(scheme="NVFP4A16", dampening_frac=0.1), ignorelm_head+embed_tokens. Calibration: 512 samples oftheblackcat102/evol-codealpaca-v1(code, disjoint from the eval sets), 2048 tokens. - Base:
openbmb/MiniCPM5-2B(LlamaForCausalLM, 2.5 B, Apache-2.0). - Built and evaluated on an RTX 5070 Ti (Blackwell, SM120), vLLM 0.26.
License
Apache-2.0, inherited from openbmb/MiniCPM5-2B.