RaceBench-MiniCPM5-v1.1
Quality over quantity — and v1.1 is the receipt.
Full-parameter fine-tune of openbmb/MiniCPM5-1B on RaceBench-v1.1 (376,736 rows: 353,456 distilled + 23,280 premium agent traces at 4x upsample).
The honest headline: v1.1 does NOT beat base. The premium-agent experiment reversed v1.0's reasoning gain (BBH -9.4, z=-12.3) while keeping a smaller math regression (GSM8K -5.7, z=-3.0). The 4x-upsampled long agent traces look harmful to a 1B model. This is a negative result, published honestly - and it cleanly redirects v1.2.
Results: RaceBench-MiniCPM5-v1.1 vs base
lm-evaluation-harness v0.4.12, identical settings head-to-head (fp16, seed 42, single NVIDIA H100). Deltas in points; z = delta / pooled SE; |z| > 2 is significant at ~95%.
| Task | v1.1 | MiniCPM5-1B (base) | Delta | z | |
|---|---|---|---|---|---|
| BBH (CoT few-shot, n=6511) | 50.94 ± 0.55 | 60.34 ± 0.53 | -9.40 | -12.3 | significant LOSS |
| GSM8K (strict-match, n=1319) | 34.34 ± 1.31 | 40.03 ± 1.35 | -5.69 | -3.0 | significant LOSS |
| MMLU (full, n=14042) | 51.78 ± 0.40 | 53.38 ± 0.40 | -1.60 | -2.8 | significant small LOSS |
| Winogrande (n=1267) | 56.27 ± 1.39 | 56.35 ± 1.39 | -0.08 | -0.04 | noise |
| HellaSwag acc_norm (n=10042) | 48.41 ± 0.50 | 48.82 ± 0.50 | -0.41 | -0.58 | noise |
| TruthfulQA MC2 (n=817) | 44.77 ± 1.49 | 45.99 ± 1.49 | -1.22 | -0.58 | noise |
| GPQA-Main zeroshot (n=448) | 27.90 ± 2.12 | 27.01 ± 2.10 | +0.89 | +0.30 | noise |
| ARC-Challenge (n=1172) | 34.98 ± 1.39 | 34.22 ± 1.39 | +0.76 | +0.39 | noise |
Reading this correctly
- Significant regressions: BBH -9.4, GSM8K -5.7, MMLU -1.6. Nothing improves significantly.
- Do not compare these absolutes to the v1.0 model card table - that run used different task configs (zero-shot BBH variant, MMLU subset, 2xT4 venue). Within THIS run, both models were evaluated identically, so deltas are apples-to-apples.
- vs v1.0 directionally: v1.0 delivered BBH +2.7 / GSM8K -9.9. v1.1 delivers BBH -9.4 / GSM8K -5.7. Adding 4x-upsampled premium traces traded the reasoning gain away for a smaller math hit. Bad trade at 1B scale.
Why this happened (hypothesis)
The premium agent traces average 46 turns / ~2.7k chars (max 135k). At 1B scale, 6.2% of training mass in very long multi-turn traces appears to have taught verbose drift, not planning. Per the pre-registered decision tree this is Branch B: ablate premium threshold (>=0.85 or >=0.90 instead of >=0.8), cap trace length (<8192 chars, <=12 messages), re-test.
Deployment implications (read before adopting)
- Do not use v1.1 over the base model today. Every significant delta is negative.
- Use
saidutta69/RaceBench-MiniCPM5(v1.0) if you want the reasoning-gain checkpoint - with its documented GSM8K cost. - IFEval/HumanEval/MBPP were attempted on H100 and are pending harness fixes; not reported here and no claims are made about instruction-following or code.
Training details
- Base model: openbmb/MiniCPM5-1B
- Dataset: saidutta69/RaceBench-v1.1
- Method: full-parameter SFT, seq_len 4096, sample packing, assistant-only loss masking
- Precision: bf16, SDPA attention, adamw_bnb_8bit
- Schedule: lr 2e-05 cosine, warmup 100, wd 0, seed 42, effective batch 8 x 4096 tokens (32k tok/step), 1 epoch = 10,445 steps, final train_loss 1.301
- Data volume: 376,736 rows -> 342.2M tokens -> 83,557 packed sequences
- Hardware: NVIDIA H100 80GB, ~3h wall
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"saidutta69/RaceBench-MiniCPM5-v1.1",
trust_remote_code=True,
torch_dtype="bfloat16",
)
tokenizer = AutoTokenizer.from_pretrained("saidutta69/RaceBench-MiniCPM5-v1.1")
License
Apache-2.0 (base model weights); RaceBench data is MIT.
Citation
@misc{racebench-minicpm5-v11,
author = {Sai Dutta},
title = {RaceBench-MiniCPM5-v1.1 -- Premium-Agent Ablation Checkpoint},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/saidutta69/RaceBench-MiniCPM5-v1.1}}
}