Qwen2.5-0.5B grounded RLOO
The RLHF policy. Qwen/Qwen2.5-0.5B-Instruct is supervised-fine-tuned on preferred answers, a reward model is trained on the preference pairs, and this policy is optimized with RLOO against that reward model.
RLOO is the online RL stage in current TRL. The policy samples two completions per prompt, the reward model scores them, and a leave-one-out baseline reduces the variance of the update. A KL penalty (beta 0.05) keeps the policy near the SFT reference. TRL 1.13 removed PPOTrainer; this repository uses the method TRL still ships for online RLHF.
The training data is derived from SQuAD 2.0 and is licensed CC BY-SA 4.0. These weights stay under the base model's Apache 2.0 license.
Prompt
Use the Qwen2.5 chat template. The system message is:
You answer a question using only the passage in the user message. Reply with a short answer taken from the passage. If the passage does not contain the answer, reply with exactly: The passage does not say.
Put the question first and the passage after it in the user message. See app.py in the training repository for a Gradio demo.
Training
- Policy init: the merged SFT model
- Reward: the merged reward model from the same run
- Trainer: TRL
RLOOTrainer, LoRA, merged before upload - Generations per prompt: 2
- The config file that produced these weights is named in the metrics section below
Evaluation
Greedy decoding. Strict accuracy requires the gold span in a short reply that does not abstain, or exactly The passage does not say. The table also reports normalized answer EM and token F1 on answerable questions, and semantic abstention on unanswerable questions. Semantic abstention is a short refusal in any wording. A refusal that also gives a span does not count. When the run uses configs/protocol.yaml, these numbers are the article-held-out test split. Trainer logs are not used to select a checkpoint. The win rate is how often this policy's reply scores above the comparison model's reply under the reward model.
resolved.yaml, environment.json, and run.json in this repository are the configuration, package versions, and wall time for the run. Full predictions are predictions.jsonl in the dataset repository.
Config: protocol.yaml.
| Model | Strict acc. | Answerable | Unanswerable | Answer EM | Answer F1 | Semantic abstention |
|---|---|---|---|---|---|---|
| base | 0.410 | 0.290 | 0.530 | 0.200 | 0.290 | 0.530 |
| always_abstain | 0.500 | 0.000 | 1.000 | 0.000 | 0.000 | 1.000 |
| sft | 0.550 | 0.310 | 0.790 | 0.300 | 0.346 | 0.790 |
| rloo | 0.475 | 0.590 | 0.360 | 0.520 | 0.616 | 0.360 |
| Comparison | Reward-model win rate | Ties |
|---|---|---|
| rloo_vs_sft | 0.410 | 0.565 |
| rloo_vs_base | 0.510 | 0.340 |
| rloo_vs_always_abstain | 0.705 | 0.260 |
| sft_vs_base | 0.285 | 0.480 |
| sft_vs_always_abstain | 0.315 | 0.670 |
Limitations
A 0.5B model trained on a few thousand derived pairs will not be a reliable reader. The reward model can be gamed, and RLOO can raise the reward without raising task accuracy. Read the metrics above before treating a win as a better answer. Not for high-stakes use.