slm-500m-qa-RLAIF
RLAIF (GRPO) applied to
thesreedath/slm-500m-qa, a QA-SFT model, to sharpen answer
quality on closed-book (question only) prompts.
Results
| metric | value |
|---|---|
| mean reward, first half of training | 3.4834 |
| mean reward, last half | 11.8394 |
| reward gain | +8.356 |
| mean KL to the frozen SFT reference | 0.568 |
300 prompts x group of 4, 75 steps, KL beta 0.04, in 275s.
How it was trained
RLAIF via GRPO. For each prompt the policy samples a group of 4 answers; a reward model trained on AI-generated preferences scores them; the advantage is each sample's reward minus the group mean, divided by the group standard deviation. GRPO rather than PPO because it needs no value network, so the policy, its frozen reference and the reward model fit together and the run stays short.
The reward model is a 125M backbone with a scalar head, trained by Bradley-Terry loss on all 600 preference pairs. It reached 93.3% accuracy on 60 held-out pairs (chance is 50%, and it measured 53.3% before training).
Preference data
600 triplets (prompt / chosen / rejected), 300 grounded and 300 closed-book. This model trained on the closed_book half, because that is the distribution it was supervised-fine-tuned on.
The rejected answer in each pair carries exactly one deliberately induced flaw, drawn evenly from six modes that these models actually exhibit: a wrong figure, an unsupported claim, vagueness, rambling repetition, a partial answer, and a false refusal. An LLM judge then confirmed the ordering with the two answers shown in randomised positions, so it could not score well by always choosing the first. Pairs the judge disagreed with, or was unconfident about, were dropped.
Prompt format
This model expects plain_qa:
Question: {question}\nAnswer:
The format was established empirically, by scoring known-good answers under competing templates and comparing generation behaviour -- not assumed from the tokenizer.
Limits
600 preference pairs is a small budget for preference optimization. Expect sharper formatting, less padding and more consistent refusals -- not new capability. Preference labels come from an LLM judge, so the model inherits that judge's blind spots. Not legal or financial advice.