RLCR-llama32-3B-bigmathdigits5000
This model is Llama-3.2-3B-Instruct trained with RLCR (Reinforcement Learning with Calibration Rewards, Damani et al. 2026) on Big-Math-digits: adds a Brier-score-based reward on verbalized confidence to the correctness reward, jointly optimizing reasoning and calibration.
Results
| Metric | Value |
|---|---|
| Accuracy (held-out) | 0.287 |
| ECE | 0.185 |
| PCE | 0.171 |
| Brier score | 0.242 |
| AUROC | 0.583 |
Table 4 of the paper. Confidence source for the Llama-3.2-3B RLCC curricula.
LoRA
LoRA target modules: PEFT architecture default for Llama (q_proj, v_proj only).
Rank 16, alpha 32, dropout 0.05.
Training details
- Algorithm: GRPO, 1 epoch, per-device batch size 1, KL penalty β = 0, AdamW (8-bit), max grad norm 1.0.
- Dataset: mehuldamani/big-math-digits, 5,000 training samples, 1,000 held-out for evaluation.
- Rollouts: vLLM, temperature 0.7, 32 generations per prompt, effective rollout batch size 1,024.
- Learning rate: 5e-6, linear schedule, warmup ratio 0.20.
- Max response length: 1,024 tokens.
- Answer judging: rule-based (exact match on extracted final answer).
- Compute: single RTX 3090 (16-36 GPU-hours) except Qwen3-4B (single A100, 13-19 GPU-hours).
Citation
This checkpoint accompanies the paper "Confidence as Curriculum: Reinforcement Learning for Joint Reasoning and Calibration" (currently under anonymous peer review). A formal citation will be added once the paper is public. It builds directly on:
@inproceedings{damani2026rlcr,
title = {Beyond Binary Rewards: Training {LM}s to Reason about their Uncertainty},
author = {Damani, Mehul and Puri, Isha and Slocum, Stewart and Shenfeld, Idan and Choshen, Leshem and Kim, Yoon and Andreas, Jacob},
booktitle = {The Fourteenth International Conference on Learning Representations},
year = {2026}
}