1PP 1.7B (unconditioned (raw)) — GRPO on GSM8K
GRPO fine-tune of Raghav-Singhal/1pp-1.7b-raw-sft
on GSM8K, 100 policy steps. Trained with verl
0.9.0.dev0 on 4x GH200.
One of three siblings that differ only in the pre-training persona condition of
the starting checkpoint — asst, ua, raw — trained with an identical recipe so
the conditions can be compared after RL.
Results
GSM8K test, 16 samples per question, temperature 1.0.
| pass@1 | pass@16 | |
|---|---|---|
| before RL (step 0) | 0.0081 | 0.0715 |
| after RL (step 100) | 0.0369 | 0.1818 |
Full curve:
| step | pass@1 | pass@16 | entropy | resp. chars |
|---|---|---|---|---|
| 10 | 0.0181 | 0.1435 | 1.305 | 356 |
| 20 | 0.0225 | 0.1470 | 1.037 | 365 |
| 30 | 0.0317 | 0.1956 | 0.723 | 337 |
| 40 | 0.0293 | 0.1702 | 0.499 | 321 |
| 50 | 0.0356 | 0.1909 | 0.401 | 292 |
| 60 | 0.0334 | 0.2046 | 0.340 | 285 |
| 70 | 0.0354 | 0.1951 | 0.281 | 297 |
| 80 | 0.0347 | 0.1687 | 0.255 | 292 |
| 90 | 0.0393 | 0.1963 | 0.231 | 292 |
| 100 | 0.0369 | 0.1818 | 0.213 | 285 |
Caveat on comparing the three siblings
A configuration-identical repeat of the asst arm reached pass@1 0.0552 at step 100
where this run reached 0.0393. That seed-to-seed gap is larger than the
spread across the three conditions (0.0369-0.0413), so the ranking between asst,
ua and raw here is not evidence of a real difference. Treat each model on its own
absolute improvement, not against its siblings, until multiple seeds exist.
Checkpoints
main is step 100. Every earlier checkpoint is a branch:
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "VityaVitalich/1pp-1.7b-raw-sft-grpo-gsm8k"
model = AutoModelForCausalLM.from_pretrained(repo) # step 100
model = AutoModelForCausalLM.from_pretrained(repo, revision="step-50") # step 50
tok = AutoTokenizer.from_pretrained(repo)
Branches: step-10, step-20, step-30, step-40, step-50, step-60, step-70,
step-80, step-90, and main (= step 100).
Training
| algorithm | GRPO (group-relative advantages, no critic), KL penalty retained |
group size (rollout.n) | 16 |
| prompts per step | 256 |
| mini-batch | 128 prompts |
ppo_epochs | 3 |
| learning rate | 1e-6, constant |
| max prompt / response | 256 / 512 tokens |
| reward | verl's stock GSM8K scorer, unmodified |
| precision | bf16 |
| hardware | 4x GH200, ~50 s/step |
ppo_epochs=3 was chosen by sweep: 1, 2, 4 and 5 all validate worse at matched steps,
and 4 and 5 reach a higher training reward while validating lower — the signature of
over-reusing a fixed batch of rollouts.
Notes
Weights are bf16. verl exports FSDP's fp32 master copy and declares bfloat16 in
config.json; the cast here makes the file agree with the config and is lossless for
inference. The tokenizer files come from the base repo unchanged — RL does not touch
the tokenizer, and verl's export omits four of them.