smollm2-360m-boolq-calibration-grpo2-v1
GRPO-trained checkpoint from slm-calibration-rlcd — a project training small LMs to make calibrated yes/no decisions, inspired by TypeSafe AI's "Jev" model / "RLCD" (Reinforcement Learning for Calibrated Decisions). Full project: 25b3nk/slm-calibration-rlcd.
This is the full GRPO-trained model, built on top of sft-warmup-v2. It is a real but modest calibration improvement, not a finished or perfectly calibrated model — see the eval table and discussion below before over-reading the result.
The interesting part: why the previous GRPO attempts failed, and what fixed it
Three earlier GRPO runs on top of the original SFT checkpoint (sft-warmup-v1) all collapsed: the model's confidence output converged to just 1-2 of the two fixed confidence-target constants used in v1's SFT data (0.92 and/or 0.65), regardless of training steps or learning rate. That's not calibration — it's the model parroting the SFT heuristic. Root cause: v1's SFT prior had near-zero output entropy on the confidence dimension, so GRPO had nothing to diversify from.
The fix was not a GRPO hyperparameter change — it was upstream, in SFT: v2 (see sft-warmup-v2) samples confidence targets from ranges instead of two fixed constants, and its prompt explicitly asks for a confidence value. This checkpoint (grpo2-v1) reuses the exact same GRPO hyperparameters as the failed v1 attempts (lr=1e-6, confidence-alignment weight λ=0.5, KL penalty β=0.04, num_generations=6, 1 epoch, 56 steps) on top of sft-warmup-v2, specifically to isolate the SFT/prompt change as the cause of the fix.
It worked: this checkpoint produced 11 distinct confidence values across the 500-example eval set (spanning roughly 0.61-0.89), versus 1-2 distinct values for every prior GRPO attempt. That's the key qualitative result — genuinely non-degenerate, varied confidence output for the first time in this project.
A correctness-only control run (grpo2-control-v1: same pipeline and hyperparameters, but the confidence-alignment reward term zeroed) was also trained for comparison. The treatment (this checkpoint) beats the control on both calibration metrics at equal accuracy, which validates that the calibration-specific reward terms are doing real, measurable work rather than just adding noise.
What it does
Given a BoolQ-style passage + yes/no question, outputs:
<answer>yes|no</answer><confidence>0.XX</confidence>
Training
- Base: HuggingFaceTB/SmolLM2-360M-Instruct, warm-started with sft-warmup-v2
- GRPO, 56 steps, 1 epoch, lr=1e-6, num_generations=6, KL penalty β=0.04
- Training set: 453-example solve-rate-filtered subset of BoolQ train
- Reward:
1[answer==label] - (confidence - 1[answer==label])^2 - λ*(confidence - p_logit)^2— correctness + Brier term + a confidence-alignment term (λ=0.5) pulling the stated confidence toward the model's own internal logit-derived probability
Eval (same 500-example BoolQ validation slice, seed 0)
| Metric | sft-warmup-v2 (pre-GRPO) | grpo2-control-v1 (correctness-only reward) | This checkpoint (grpo2-v1) |
|---|---|---|---|
| Format compliance | 1.0 | 1.0 | 1.0 |
| Accuracy | 0.804 | 0.79 | 0.792 |
| Brier score | 0.167 | 0.176 | 0.173 |
| ECE | 0.090 | 0.098 | 0.086 |
| Distinct confidence values | n/a | not measured for this comparison | 11 (spanning ~0.61-0.89) |
Two things worth being precise about, not glossing over:
- Both GRPO runs (this checkpoint and the control) actually have a higher Brier score than sft-warmup-v2 alone (0.173 / 0.176 vs 0.167) — GRPO on top of this SFT stage is not a strict win over the SFT-only checkpoint on every metric. The "11 distinct confidence values" figure is compared against the earlier, SFT-v1-based GRPO attempts (which produced only 1-2 distinct values), not against the control, which shares SFT v2's higher-entropy prior and was not measured for this specific statistic.
- Calibration (ECE 0.086) is only a modest improvement over sft-warmup-v2's own ECE (0.090) — this is not a fully calibrated model.
The result to take away is qualitative as much as quantitative: for the first time in this project, GRPO produced varied, non-degenerate confidence output instead of collapsing to 1-2 constants (as every prior GRPO attempt on top of sft-warmup-v1 did), and this checkpoint beat a correctness-only control trained on the identical pipeline on both calibration metrics (ECE, Brier) at essentially equal accuracy (0.792 vs 0.79) — evidence the calibration-specific reward terms are doing real, measurable work.
Full design + ADRs
(No GRPO-specific ADR exists yet for this run at push time; see the project repo's tracking notes / experiment logs for the full run details referenced above.)