1PP 1.7B (assistant-conditioned) — GRPO on GSM8K
GRPO fine-tune of Raghav-Singhal/1pp-1.7b-asst-sft
on GSM8K, 100 policy steps. Trained with verl
0.9.0.dev0 on 4x GH200.
One of three siblings that differ only in the pre-training persona condition of
the starting checkpoint — asst, ua, raw — trained with an identical recipe so
the conditions can be compared after RL.
Results
GSM8K test, 16 samples per question, temperature 1.0.
| pass@1 | pass@16 | |
|---|---|---|
| before RL (step 0) | 0.0110 | 0.0979 |
| after RL (step 100) | 0.0393 | 0.2008 |
Full curve:
| step | pass@1 | pass@16 | entropy | resp. chars |
|---|---|---|---|---|
| 10 | 0.0203 | 0.1573 | 1.281 | 399 |
| 20 | 0.0225 | 0.1705 | 1.123 | 336 |
| 30 | 0.0315 | 0.2268 | 0.769 | 287 |
| 40 | 0.0286 | 0.1722 | 0.559 | 249 |
| 50 | 0.0376 | 0.2155 | 0.451 | 231 |
| 60 | 0.0332 | 0.2097 | 0.388 | 225 |
| 70 | 0.0312 | 0.1958 | 0.358 | 213 |
| 80 | 0.0422 | 0.2110 | 0.339 | 206 |
| 90 | 0.0352 | 0.1957 | 0.319 | 212 |
| 100 | 0.0393 | 0.2008 | 0.293 | 223 |
Caveat on comparing the three siblings
A configuration-identical repeat of the asst arm reached pass@1 0.0552 at step 100
where this run reached 0.0393. That seed-to-seed gap is larger than the
spread across the three conditions (0.0369-0.0413), so the ranking between asst,
ua and raw here is not evidence of a real difference. Treat each model on its own
absolute improvement, not against its siblings, until multiple seeds exist.
Checkpoints
main is step 100. Every earlier checkpoint is a branch:
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "VityaVitalich/1pp-1.7b-asst-sft-grpo-gsm8k"
model = AutoModelForCausalLM.from_pretrained(repo) # step 100
model = AutoModelForCausalLM.from_pretrained(repo, revision="step-50") # step 50
tok = AutoTokenizer.from_pretrained(repo)
Branches: step-10, step-20, step-30, step-40, step-50, step-60, step-70,
step-80, step-90, and main (= step 100).
Training
| algorithm | GRPO (group-relative advantages, no critic), KL penalty retained |
group size (rollout.n) | 16 |
| prompts per step | 256 |
| mini-batch | 128 prompts |
ppo_epochs | 3 |
| learning rate | 1e-6, constant |
| max prompt / response | 256 / 512 tokens |
| reward | verl's stock GSM8K scorer, unmodified |
| precision | bf16 |
| hardware | 4x GH200, ~50 s/step |
ppo_epochs=3 was chosen by sweep: 1, 2, 4 and 5 all validate worse at matched steps,
and 4 and 5 reach a higher training reward while validating lower — the signature of
over-reusing a fixed batch of rollouts.
Notes
Weights are bf16. verl exports FSDP's fp32 master copy and declares bfloat16 in
config.json; the cast here makes the file agree with the config and is lossless for
inference. The tokenizer files come from the base repo unchanged — RL does not touch
the tokenizer, and verl's export omits four of them.