Qwen3-4B-Instruct-2507.prestar-RL.reason-only.lr7e-7-kl0.step-2176
Qwen/Qwen3-4B-Instruct-2507 fine-tuned with RL to write short "thoughts" about the text that immediately follows a
cut in a stat.ML arXiv paper. This is the step 2176 policy of the prestar-RL run
reason-only / stride-1 consumer / lr 7e-7, no KL, exported in fp32 from the trainer's FSDP master
weights. It is the best-validation checkpoint of the run at the time of export.
What it was trained to do
Given a document prefix, the policy writes a few dense sentences inside <thought> … </thought> about the
next 8 tokens. The reward is the mean per-token log-probability of the true 8-token continuation under a
frozen suffix conditional, JackHsieh/4B-base.4B-Instruct-reason-thoughts.stride-1.suffix-conditional, which reads the thought in a
KEY/VALUE hint wrap prepended to the document. Higher is better; the base model scores -1.1493
nats/token on the validation grid and this checkpoint scores -1.1420.
Training
- Algorithm: RLOO (leave-one-out baseline), 8 samples per prompt, 64 prompts per step, CISPO-style policy loss, no KL term
- Optimizer: AdamW, constant lr 7e-7
- Prompts: the first 1,024 documents of
JackHsieh/statML-arxiv-RL-4k-docs, every 8-token chunk (stride 1) - Sampling during training: temperature 0.6, top-p 0.9, stop at
</thought>, 1,024-token cap; a rollout with no</thought>receives a fixed penalty of -8 nats/token - Validation: one sample per chunk on the stride-32 grid of the
JackHsieh/statML-arxiv-40M-20Mtest split (72,960 chunks) - Framework: verl with vLLM rollouts; 8x H200
- Logs: wandb run (the run was requeued several times; each segment is a separate wandb run in the same group)
Usage
Prompt with the reason-only chat template used for training: a system message describing the task, the raw
document prefix as the user turn, and an assistant turn pre-filled with <thought>\n. Sample at temperature 0.6,
top-p 0.9, and stop on </thought>. The template is in the prestar repository (chat_templates/qwen/reason-only-nothink.jinja).