seeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp5_vs16 — RL checkpoint (global step 4)
Role in this run: best by pass@8
What it is
An OpenRLHF GRPO reinforcement-learning checkpoint for Qwen3-4B.
- Base model:
Qwen/Qwen3-4B-Instruct-2507 - Seeded from: base Qwen3-4B (no SFT seed — RL applied directly to the base model)
- Saved at global step 4 of RL run
seeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp5_vs16.
What it's best at
- This is the best checkpoint by pass@8 so far in this run.
Trained and validated on the cobalt-train ≤2/64 frontier (canonical clean_eval prompts): 1833 train / 112 held-out val problems the base model solved on at most 2 of 64 samples under the iid_canonical@64 hardness scan. Val evals sample at temperature 1.0 (matching the clean_eval frontier eval).
Reward signal: binary code-correctness (1.0 if the generated program passes the problem's tests, otherwise 0.0).
Eval metrics at this checkpoint: not available in the train log.
Where the logs are
- Weights & Biases: project
eaiexp-paper-final, run nameseeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp5_vs16. - Local train log:
experiments/cobalt_qwen3_4b_ft/rl_runs/qwen3_4b_instruct_2507_cobalt_v1/seeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp5_vs16/openrlhf_train.log
Recipe
- Algorithm: GRPO (group-normalized advantages, no KL penalty).
- Stop-properly penalty: truncated samples' reward set to -1.0 (ProRL-style anti-truncation shaping).
- DAPO overlong penalty: responses in the last 1024 tokens before the cap get an additive penalty ramping to -0.25.
- Samples per prompt: 8
- Rollout batch size: 128, train batch size: 128
- Max new tokens per rollout: 4096
- Episodes: 2
- Actor learning rate: 1e-06 (constant schedule)
How to load
This checkpoint is the main revision (git branch) of the repo, with the model at the repo root — load it directly, no subfolder:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("agurung/cobalt-seeded-rl-base-ramp25-stoppen-gen4k-ep2-ncp5-vs16", revision="main")
tokenizer = AutoTokenizer.from_pretrained("agurung/cobalt-seeded-rl-base-ramp25-stoppen-gen4k-ep2-ncp5-vs16", revision="main")
Or serve with vLLM: vllm serve agurung/cobalt-seeded-rl-base-ramp25-stoppen-gen4k-ep2-ncp5-vs16 --revision main