Qwen3-8B-GRPO-MATH
Qwen3-8B trained with GRPO on the Hendrycks MATH training split, using verifiable final-answer rewards. Produced as the RL teacher for a study on how much of a teacher's RL gain survives on-policy distillation into a smaller student.
This is a short-budget research checkpoint, not a tuned release. It is 120 optimizer updates over roughly a quarter of one epoch. See Honest limitations below before using it.
Training
| Base model | Qwen/Qwen3-8B |
| Algorithm | GRPO (group-relative advantage, no critic) |
| Framework | slime - SGLang rollout + Megatron training + Ray |
| Data | EleutherAI/hendrycks_math train split, 7,496 prompts after dropping 4 with no parseable oxed{} answer |
| Reward | Rule-based final-answer equivalence (slime --rm-type math) |
| Optimizer updates | 120 |
| Rollout batch | 16 prompts/step x 8 samples = 128 |
| Global batch | 128 (so 1 update per rollout step) |
| Learning rate | 1e-6, constant |
| Max response length | 4,096 tokens |
| Chat mode | non-thinking (enable_thinking=false) |
| Precision | bf16, --use-precision-aware-optimizer --optimizer-cpu-offload |
| Hardware | 1x H100 NVL (93 GiB), ~2 h |
Prompt coverage is 16 x 120 = 1,920 of 7,496 prompts, i.e. 0.26 epoch.
Reward-function caveat worth repeating
slime ships three math scorers and only one is correct for these labels. Verified on 300 real MATH labels by constructing responses that are correct by definition:
--rm-type | true positives | false positives |
|---|---|---|
math | 300/300 | 0/300 |
dapo | 202/300 | 0/300 |
deepscaler | 0/300 | 0/300 |
deepscaler splits on </think> and returns 0 for every non-thinking response;
dapo needs a literal Answer: line. Only ~64% of MATH labels are numeric, the rest are
symbolic LaTeX, and math handles all 94 symbolic labels in the sample. This model was
trained with --rm-type math.
Evaluation
avg@16 / pass@16, non-thinking chat mode, temperature 1.0, top-p 1.0, no top-k, scored with the same harness for every row.
AIME 2024 + 2025, at a 16,384-token budget where truncation is 0-1% and the headline number is no longer budget-limited:
| avg@16 | pass@16 | mean length | trunc% | |
|---|---|---|---|---|
| Qwen3-8B (base) | 22.09 | 46.7 | 2,325 | 0.0 |
| this model | 34.27 | 66.7 | 5,203 | 0.6 / 1.0 |
MATH-500, at a 4,096-token budget:
| avg@16 | pass@16 | |
|---|---|---|
| Qwen3-8B (base) | 80.84 | 92.8 |
| this model | 85.50 | 94.6 |
pass@16 rises alongside avg@16 on every benchmark, so this run expanded the set of solvable problems rather than only sharpening the top-1 answer.
Read the budget before you read the score
The model roughly doubles its response length (2,325 -> 5,203 tokens), and a truncated response scores exactly zero. The same checkpoint measured at smaller budgets:
| AIME 24/25 avg@16 | 4,096 | 8,192 | 16,384 |
|---|---|---|---|
| score | 28.96 | 34.06 | 34.27 |
| truncation | 48.5% / 46.0% | 21.0% / 17.7% | 0.6% / 1.0% |
At 4,096 tokens you are largely measuring how often it finishes, not whether it can solve the problem. Give it 16k.
Honest limitations
- Short budget. 120 updates, 0.26 epoch of the 7,496 prompts. Not a converged RL run.
- Weak gradient signal. MATH is easy for Qwen3-8B: mean rollout reward ran 0.65-0.97, and 12-13 of every 16 prompt groups had zero reward variance, so ~78% of rollout compute produced no gradient. A harder prompt set or DAPO-style dynamic sampling would help.
- Length inflation. See the budget table above. This is the single easiest way to mis-measure this model.
- The training response cap matters enormously. An otherwise identical run with the rollout cap at 2,048 instead of 4,096 tokens reached only +2.04 overall and lost a point on MATH-500. The cap, not the step count, was the binding constraint.
- Single seed. AIME24/25 are 30 problems each; repeated avg@16 evaluations of the same model differ by about +/-2 points. Treat any difference under 2 points as noise.
Intended use
Research on RL-for-reasoning and on-policy distillation. Not intended for deployment.
License
Apache-2.0, inherited from Qwen3-8B. The MATH dataset is MIT-licensed.