slm-125m-legal-dpo-v2
Direct Preference Optimization on top of
abhishekai/slm-125m-legal-sft-v2,
for grounded question answering over legal and financial passages.
This is the best 125M checkpoint in the project, and the first RLAIF result here that an independent judge confirms rather than reverses.
The result
| mean /10 | qa | instr. following | grounding | refusal | |
|---|---|---|---|---|---|
sft-v2 (its own baseline) | 4.13 | 3.18 | 3.29 | 4.52 | 5.55 |
dpo-v2 | 4.82 [4.48, 5.17] | 3.85 | 3.77 | 5.71 | 5.96 |
Paired over the same 300 prompts: +0.69, bootstrap 95% CI [+0.34, +1.04], 156 W / 112 L / 32 T. The interval excludes zero, so this is a measured improvement, not a point estimate that happens to be positive. Every axis rose; grounding most (+1.19).
Repetition collapse also more than halved, 4.3% → 1.7% — which matters more for a served model than a judged one.
Why the preference data is different from this project's earlier RLAIF runs
Two defects made every previously published DPO/PPO number here uninformative, and both are fixed in this checkpoint:
- Prompts come from a reserved pool that is disjoint from the SFT training and eval splits, asserted at load time. The earlier runs drew preference prompts from the SFT training set, so candidates were sampled on questions the policy had memorised and the ranker was separating near-identical answers.
- Candidates are sampled on-policy from
sft-v2itself, the model being improved.
The funnel: 1,194 prompts → 758 ranked pairs → 725 confirmed by an independent pairwise check → 625 train / 100 eval.
A caveat about how RLAIF models here are usually scored
Its Gemini-judged win-rate is 0.545 at n=100, which is not significant (p = 0.349). That number is not the evidence for this model, and the reason is worth stating plainly: Gemini ranked the preference pairs and judged the win-rate, so it was asked whether a policy trained toward its own taste matched its own taste.
Scored instead by Claude Sonnet — a different model family, paired over 3× the prompts on an absolute scale — the same comparison gives +0.69 with an interval excluding zero. On the identical run, Gemini scored a Gemma DPO checkpoint at the same 0.545 while Sonnet found it had regressed by −0.97. Two identical-looking win-rates, opposite outcomes.
Prefer the absolute scores above to any win-rate, including this model's own.
How it was trained
DPO, 114 steps, held-out DPO loss 0.6716, preference accuracy 0.865, margin 0.654. 1×H100, 5.8 minutes, $0.38. Best-checkpoint selection is on held-out DPO loss, not preference accuracy — an earlier run reached 0.96 accuracy at a 0.455 win-rate, so accuracy is not the quantity that tracks quality here.
It did not converge. The best evaluation was the final step and early stopping never fired, so 0.6716 is a floor for this recipe rather than its ceiling.
Prompt format
It expects a chat template with a system prompt, and a user turn containing the question
followed by Context: and the passage. Use tokenizer.apply_chat_template.
How it was evaluated
Claude Sonnet as an independent judge — a different model family from the Gemini 2.5 Flash that wrote the training data and ranked the preference pairs. Four axes out of 10 (question answering, instruction following, grounding in the supplied passage, appropriate refusal), scored against the prompt, the answer, a golden answer and the corpus evidence. The same 300 held-out prompts score every model in this project.
The judge was gated before use: it had to reproduce a known ordering (base Gemma-2-2B above a broken fine-tune of it) before any of its scores were trusted. It did, at p = 0.0022.
Limitations
- It answers from a passage you supply. Remove the context and it confabulates fluently. This is a reading-comprehension model for a retrieval pipeline, not a knowledge store.
- ~1.7% of outputs still degenerate into repetition.
- No arithmetic reliability. It will state figures and a percentage change that do not follow from each other. Expected at this scale.
- It invents citations. It reproduces the register of judicial opinions convincingly, so invented case names and holdings look plausible.
- 4.82/10 is a low absolute score. It is the best 125M here, and it is still a small model.
- Not legal or financial advice. It is a demonstration of method.
- One judge, one eval set, n=300. The eval supplies the passage, which favours this style of model. No labelled benchmark (CaseHOLD, LexGLUE) has been run, so nothing here says the model is good in absolute terms.