delphi-1e22-p33m67 · xorl GRPO reproduction of rlvr7500_w1 (step 146)
Dense Qwen3 ~9.7B trained with forced-thinking GRPO math RL in
xorl server-mode — a cross-framework reproduction of
marin issue #6279's rlvr7500_w1 run
(originally trained with MarinSkyRL). Same start checkpoint
(wc386k_lr1e5-sft),
same data (RLVR-MATH-7500), same recipe (boxed-answer verifier + length penalty lpw=1.0,
no KL / no entropy reg, lr 1e-5, grad clip 0.5, bs 256 × G=16, 146 steps, 4k budget,
temp 0.7 / top_p 1.0) — trained fully on-policy on 16 H100s (8 trainer + 8 samplers) in 8.0 h.
Results (same-harness ΔSFT; reference = rlvr7500_w1 step 140)
| metric | SFT base | this model | Δ | reference Δ |
|---|---|---|---|---|
| MATH500 | 48.69 | 60.97 | +12.28 | +8.4 |
| gsm8k-flex | 65.28 | 74.07 | +8.79 | +4.4 |
| AIME24 (10 seeds × 300) | ~2.6–3.1 | 3.77 ± 0.47 | ≈ +1 (small/noisy) | −3.4 |
Deltas are measured with one harness against its own SFT baseline on each side; absolute numbers across harnesses are not comparable. AIME baseline is partial (9/10 seeds) — the robust claim is "did not regress", unlike the reference (CoT truncation at the 4k ceiling; this policy's completions stay ~450 tokens).
Training curves (public): our run ·
reference trajectory ·
timing.
Recipe code: xorl-client experiments/marin.
Usage
Standard Qwen3ForCausalLM checkpoint (bf16 safetensors). Prompts should be rendered through
the bundled chat template with the forced-thinking prefill <|start_think|>\n appended after
add_generation_prompt=True (the recipe's prompts.py does this); sample at temperature 0.7 /
top_p 1.0 within a 512 + 3584 token budget, and extract answers as Answer: \boxed{...}.
This is the final (step-146) policy — the exact weights behind the evals above.