slm-500m-legal-dpo
A 517.8M-parameter grounded legal/financial Q&A model, preference-tuned with DPO (Direct Preference Optimization) on AI-generated feedback (RLAIF).
Honest headline
This model performs at parity with the SFT model it was trained from.
| win-rate vs slm-500m-legal-sft | 0.480 |
| decisive comparisons | 29 wins / 33 losses / 38 ties (n=100) |
| two-sided binomial p | 0.704 |
| 95% CI on decisive win-rate | [0.344, 0.592] |
The confidence interval straddles 0.5. DPO here did not measurably improve the model, and did not measurably harm it either. Published as a reproducible RLAIF artifact, not as an upgrade. If you want the better-understood model, use the SFT one.
Independent absolute score (2026-08-15)
Scored by Claude Sonnet on four axes out of 10 — a different model family from the Gemini judge behind the win-rate above, and one that had no hand in writing this project's training data. 300 held-out prompts, the same prompts and the same scale used for all twelve checkpoints, so this number is comparable across models in a way no win-rate here is.
| mean score | 5.08 / 10 |
| 95% CI (bootstrap over per-prompt scores) | [4.75, 5.41] |
| paired step | +0.42 against its own SFT (4.65), CI [+0.10, +0.75] - a real improvement, and the best score any model in this project achieves below Gemma scale. |
The full twelve-checkpoint table, with every stage-to-stage interval, is in
MODEL_INDEX.md in the project repository.
Why publish a parity result
Because it is part of a controlled 3-model scaling comparison, run with an identical pipeline (same prompt pool, same Gemini 2.5 Flash judge, same preference-mining thresholds, same DPO recipe):
| model | params | DPO win-rate vs its own SFT |
|---|---|---|
| 125M | 126M | 0.455 (degraded) |
| 500M (this) | 518M | 0.480 (parity) |
| Gemma-2-2B | 2,614M | 0.530 (parity, trending positive) |
DPO's win-rate rises monotonically with model scale, crossing from harmful to parity to slightly positive. No individual point is statistically distinguishable from 0.5 at n=100 — the trend is the finding, not any single number. Distinguishing a 0.53 effect from parity would need roughly 400–600 eval prompts.
Preference data
994 triplets (894 train / 100 held-out eval), built on-policy:
- Sample 4 candidate answers from the frozen SFT model (temp 0.9, top-p 0.95)
- Gemini 2.5 Flash scores each 1–10 on correctness and grounding
- Keep the best/worst pair only if the score gap is ≥ 2
- An independent pairwise judge re-checks that
chosenreally beatsrejected, with A/B order randomized to cancel position bias - Embedding dedup on prompts, then split
Funnel: 1,498 prompts → 1,380 with ≥2 distinct candidates → 1,048 cleared the score gap (76%) → 1,011 confirmed by the independent judge (96.5%) → 994 after dedup. That 96.5% agreement rate was the highest of the three models.
Training
DPO (Rafailov et al. 2023) with an RPO-style NLL anchor on the chosen response. beta 0.1, LR 5e-6, 2 epochs, effective batch 16, 110 steps, 1×H100, 12.1 min, $0.80. Final eval preference accuracy 0.717, eval margin 0.689.
Usage
Identical to the SFT model — grounded prompts only, greedy or low temperature. See slm-500m-legal-sft.
Limitations
All of the SFT model's limitations apply (grounded-only, no arithmetic reliability, synthetic training data, 1,024-token context, not legal or financial advice), plus:
- Preference labels come from an LLM judge, not humans. They inherit Gemini 2.5 Flash's biases.
- Optimized for judged answer quality on one prompt distribution; no broader benchmark evaluation was run.
License
Apache-2.0, inherited from thesreedath/slm-500m-base via slm-500m-legal-sft.