1PP 1.7B (user-assistant-conditioned) — GRPO on GSM8K
GRPO fine-tune of Raghav-Singhal/1pp-1.7b-ua-sft
on GSM8K, 100 policy steps. Trained with verl
0.9.0.dev0 on 4x GH200.
One of three siblings that differ only in the pre-training persona condition of
the starting checkpoint — asst, ua, raw — trained with an identical recipe so
the conditions can be compared after RL.
Results
GSM8K test, 16 samples per question, temperature 1.0.
| pass@1 | pass@16 | |
|---|---|---|
| before RL (step 0) | 0.0098 | 0.0882 |
| after RL (step 100) | 0.0413 | 0.2333 |
Full curve:
| step | pass@1 | pass@16 | entropy | resp. chars |
|---|---|---|---|---|
| 10 | 0.0181 | 0.1443 | 1.238 | 445 |
| 20 | 0.0300 | 0.1924 | 0.868 | 435 |
| 30 | 0.0305 | 0.2153 | 0.598 | 443 |
| 40 | 0.0283 | 0.1817 | 0.422 | 466 |
| 50 | 0.0352 | 0.2012 | 0.347 | 440 |
| 60 | 0.0334 | 0.2047 | 0.308 | 421 |
| 70 | 0.0327 | 0.1971 | 0.276 | 438 |
| 80 | 0.0405 | 0.2263 | 0.254 | 449 |
| 90 | 0.0393 | 0.2367 | 0.236 | 445 |
| 100 | 0.0413 | 0.2333 | 0.215 | 454 |
Caveat on comparing the three siblings
A configuration-identical repeat of the asst arm reached pass@1 0.0552 at step 100
where this run reached 0.0393. That seed-to-seed gap is larger than the
spread across the three conditions (0.0369-0.0413), so the ranking between asst,
ua and raw here is not evidence of a real difference. Treat each model on its own
absolute improvement, not against its siblings, until multiple seeds exist.
Checkpoints
main is step 100. Every earlier checkpoint is a branch:
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "VityaVitalich/1pp-1.7b-ua-sft-grpo-gsm8k"
model = AutoModelForCausalLM.from_pretrained(repo) # step 100
model = AutoModelForCausalLM.from_pretrained(repo, revision="step-50") # step 50
tok = AutoTokenizer.from_pretrained(repo)
Branches: step-10, step-20, step-30, step-40, step-50, step-60, step-70,
step-80, step-90, and main (= step 100).
Training
| algorithm | GRPO (group-relative advantages, no critic), KL penalty retained |
group size (rollout.n) | 16 |
| prompts per step | 256 |
| mini-batch | 128 prompts |
ppo_epochs | 3 |
| learning rate | 1e-6, constant |
| max prompt / response | 256 / 512 tokens |
| reward | verl's stock GSM8K scorer, unmodified |
| precision | bf16 |
| hardware | 4x GH200, ~50 s/step |
ppo_epochs=3 was chosen by sweep: 1, 2, 4 and 5 all validate worse at matched steps,
and 4 and 5 reach a higher training reward while validating lower — the signature of
over-reusing a fixed batch of rollouts.
Notes
Weights are bf16. verl exports FSDP's fp32 master copy and declares bfloat16 in
config.json; the cast here makes the file agree with the config and is lossless for
inference. The tokenizer files come from the base repo unchanged — RL does not touch
the tokenizer, and verl's export omits four of them.