seeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp5_q4v3_iid16 — RL checkpoint (global step 16)
Role in this run: best by pass@8
What it is
An OpenRLHF GRPO reinforcement-learning checkpoint for Qwen3-4B.
- Base model:
Qwen/Qwen3-4B-Instruct-2507 - Seeded from: base Qwen3-4B (no SFT seed — RL applied directly to the base model)
- Saved at global step 16 of RL run
seeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp5_q4v3_iid16.
What it's best at
- This is the best checkpoint by pass@8 so far in this run (
eval_default_pass8= 0.5564).
Trained and validated on the cobalt-train ≤2/64 frontier (canonical clean_eval prompts): 1833 train / 112 held-out val problems the base model solved on at most 2 of 64 samples under the iid_canonical@64 hardness scan. Val evals sample at temperature 1.0 (matching the clean_eval frontier eval).
Reward signal: binary code-correctness (1.0 if the generated program passes the problem's tests, otherwise 0.0).
Eval metrics at this checkpoint (held-out val, 8 samples/problem):
- pass@1 (unbiased; mean per-problem fraction correct):
-0.1937 - pass@8 (problem counted solved if any sample is correct):
0.5564
Where the logs are
- Weights & Biases: project
eaiexp-paper-final, run nameseeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp5_q4v3_iid16. - Local train log:
experiments/cobalt_qwen3_4b_ft/rl_runs/qwen3_4b_instruct_2507_cobalt_v1/seeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp5_q4v3_iid16/openrlhf_train.log
Recipe
- Algorithm: GRPO (group-normalized advantages, no KL penalty).
- Stop-properly penalty: truncated samples' reward set to -1.0 (ProRL-style anti-truncation shaping).
- DAPO overlong penalty: responses in the last 1024 tokens before the cap get an additive penalty ramping to -0.25.
- Samples per prompt: 8
- Rollout batch size: 128, train batch size: 128
- Max new tokens per rollout: 4096
- Episodes: 2
- Actor learning rate: 1e-06 (constant schedule)
How to load
This checkpoint is the main revision (git branch) of the repo, with the model at the repo root — load it directly, no subfolder:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("agurung/cobalt-seeded-rl-base-ramp25-stoppen-gen4k-ep2-ncp5-q4v3-iid16", revision="main")
tokenizer = AutoTokenizer.from_pretrained("agurung/cobalt-seeded-rl-base-ramp25-stoppen-gen4k-ep2-ncp5-q4v3-iid16", revision="main")
Or serve with vLLM: vllm serve agurung/cobalt-seeded-rl-base-ramp25-stoppen-gen4k-ep2-ncp5-q4v3-iid16 --revision main