RaceBench-MiniCPM5
Full-parameter fine-tune of openbmb/MiniCPM5-1B on RaceBench, released as the reference checkpoint for the dataset - with fp16 weights and all GGUF quantizations (Q2_K through F16).
The honest headline: RaceBench transfers real multi-step reasoning gains to a 1B model (BBH +2.7, z=3.5), at the cost of a real, larger regression in math (GSM8K -9.9, z=-5.3). This is a tradeoff, not a win - and the tradeoff is the point. v2 of RaceBench adds formal-math data to close the math gap.
Results: RaceBench-MiniCPM5 vs base
Evaluated with lm-evaluation-harness v0.4.12, identical settings for both models (fp16, fixed seed, 2x Tesla T4). Standard errors reported per task; z = delta / pooled SE. |z| > 2 is statistically significant at ~95% confidence.
| Task | RaceBench-MiniCPM5 | MiniCPM5-1B (base) | Delta | z |
|---|---|---|---|---|
| BBH (zero-shot, 27 subtasks, n=6511) | 0.3397 ± 0.0056 | 0.3123 ± 0.0055 | +2.74 | +3.5 |
| GSM8K (n=1319) | 0.3108 ± 0.0127 | 0.4102 ± 0.0135 | -9.93 | -5.3 |
| MMLU (subset, n=5700) | 0.5310 ± 0.0060 | 0.5497 ± 0.0059 | -1.87 | -2.2 |
| Minerva MATH-500 (n=500) | 0.2140 ± 0.0184 | 0.2620 ± 0.0197 | -4.80 | -1.8 |
| Winogrande (n=1267) | 0.5391 ± 0.0140 | 0.5620 ± 0.0139 | -2.29 | -1.2 |
| HellaSwag (n=10042) | 0.4820 ± 0.0050 | 0.4881 ± 0.0050 | -0.61 | -0.9 |
| TruthfulQA MC2 (n=817) | 0.4710 ± 0.0149 | 0.4597 ± 0.0149 | +1.13 | +0.5 |
| GPQA Main n-shot (n=448) | 0.2589 ± 0.0207 | 0.2723 ± 0.0211 | -1.34 | -0.5 |
| ARC-Challenge (n=1176) | 0.3746 ± 0.0141 | 0.3831 ± 0.0142 | -0.85 | -0.4 |
What is and isn't significant
- Significant (|z| > 2): BBH +2.7 (reasoning gain), GSM8K -9.9 (math loss), MMLU -1.9 (knowledge loss)
- Not significant (|z| < 2): TruthfulQA, Winogrande, ARC, HellaSwag, MATH-500, GPQA - deltas within noise; no claim is made on these tasks
- Bottom line: RaceBench teaches multi-step reasoning (BBH) at the expense of math (GSM8K) and some knowledge recall (MMLU). The TruthfulQA and GPQA differences sometimes cited for this model are not statistically supported and should not be read as gains.
Deployment implications (read before adopting)
- Agents doing numeric reasoning (arithmetic, unit conversion, tool-call args): expect worse performance than the base model. GSM8K is a direct proxy; the -9.9 is real and large.
- Reasoning-heavy, math-light workloads (multi-hop analysis, planning, classification): expect measurable gains - BBH +2.7 at z=3.5 is the most robust positive signal in this eval.
- Not evaluated: instruction-following (IFEval) and code generation (HumanEval, MBPP) were infeasible on T4 and are not reported. This checkpoint should not be marketed as an "edge agent" model until those are measured.
Evaluation notes
- MMLU is a 5.7K-sample subset (100 per subtask) due to T4 memory limits; identical subset for both models. It is not comparable to published full-MMLU scores of other 1B models.
- Same seed, order, harness version for both models - deltas are head-to-head apples-to-apples, but all claims above require the significance test in the table.
- Single seed, single run per model. Significant effects (BBH, GSM8K, MMLU) survive the test; the rest should be treated as unknown, not as "retained competence."
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"saidutta69/RaceBench-MiniCPM5",
trust_remote_code=True,
torch_dtype="float16",
)
tokenizer = AutoTokenizer.from_pretrained("saidutta69/RaceBench-MiniCPM5")
GGUF quants are in gguf/ (F16, Q8_0, Q6_K, Q5_K_M, Q5_K_S, Q5_0, Q4_K_M, Q4_K_S, Q4_0, Q3_K_M, Q3_K_S, Q2_K).
Training details
- Base model: openbmb/MiniCPM5-1B (Apache-2.0)
- Dataset: saidutta69/RaceBench (agent traces + quality-filtered coding/security/distilled)
- Method: full-parameter fine-tuning, fp16
License
Apache-2.0 (base model weights); RaceBench data is MIT.
Citation
@misc{racebench-minicpm5,
author = {Sai Dutta},
title = {RaceBench-MiniCPM5 -- First Public Fine-Tune on RaceBench},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/saidutta69/RaceBench-MiniCPM5}}
}