slm-125m-legal-dpo
A 125.8M-parameter grounded legal/financial Q&A model, preference-tuned with DPO (Direct Preference Optimization) on AI-generated feedback (RLAIF).
Honest headline
This model is, if anything, slightly worse than the SFT model it was trained from. Use slm-125m-legal-sft instead.
| win-rate vs its SFT baseline | 0.455 |
| tally | 37 wins / 46 losses / 17 ties (n=100) |
| two-sided sign test on the 83 decisive pairs | p = 0.380 |
| 95% CI on the overall win-rate | [0.361, 0.552] |
The interval straddles 0.5, so the apparent degradation is not statistically established either — the honest reading is "no measurable gain, with the point estimate on the wrong side." It is published as a reproducible RLAIF artifact and as one point in a scaling comparison, not as an upgrade.
Independent absolute score (2026-08-15)
Scored by Claude Sonnet on four axes out of 10 — a different model family from the Gemini judge behind the win-rate above, and one that had no hand in writing this project's training data. 300 held-out prompts, the same prompts and the same scale used for all twelve checkpoints, so this number is comparable across models in a way no win-rate here is.
| mean score | 3.52 / 10 |
| 95% CI (bootstrap over per-prompt scores) | [3.23, 3.82] |
| paired step | -0.52 against its own SFT (4.05), CI [-0.86, -0.18] - a real regression on this scale too. |
The full twelve-checkpoint table, with every stage-to-stage interval, is in
MODEL_INDEX.md in the project repository.
Why publish a negative result
Because it is the smallest point in a controlled 3-model comparison run through an identical pipeline — same prompt pool, same Gemini 2.5 Flash judge, same preference-mining thresholds, same DPO recipe:
| model | params | DPO win-rate vs its own SFT | p |
|---|---|---|---|
| 125M (this) | 126M | 0.455 | 0.380 |
| 500M | 518M | 0.480 | 0.704 |
| Gemma-2-2B | 2,614M | 0.530 | 0.545 |
DPO's win-rate rises monotonically with model scale, crossing from harmful to parity to slightly positive. No individual point is statistically distinguishable from 0.5 at n=100 — the trend is the finding, not any single number. Distinguishing a 0.53 effect from parity would need roughly 400–600 eval prompts.
Why it does not help at 125M
The objectives optimized correctly — DPO preference accuracy reached 0.96 on held-out pairs, and the reward model trained from the same data reached 0.837 pairwise accuracy. This is a capacity finding, not a bug.
SFT-greedy decoding is already close to this model's ceiling. Distilling toward a
Gemini-class target pushes it into a fluent prose register it does not have the
capacity to fill with substance: the answers get smoother, not more correct. A
variant trained on Gemini-authored chosen responses instead of on-policy
ones was worse still (0.335, 29 wins / 62 losses), which is the same effect
seen more sharply.
Preference data
888 triplets (788 train / 100 held-out eval), built on-policy:
- Sample 4 candidate answers from the frozen SFT model (temp 0.9, top-p 0.95)
- Gemini 2.5 Flash scores each 1–10 on correctness and grounding
- Keep the best/worst pair only if the score gap is ≥ 2
- An independent pairwise judge re-checks that
chosenreally beatsrejected, with A/B order randomized to cancel position bias - Embedding dedup on prompts (cosine ≥ 0.97), then split
Prompt format
Use the chat template shipped with the tokenizer. The assistant turn ends on
<|eos|>.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("abhishekai/slm-125m-legal-dpo")
model = AutoModelForCausalLM.from_pretrained("abhishekai/slm-125m-legal-dpo")
msgs = [
{"role": "system", "content": "You are a precise legal and financial assistant. Answer only from the provided context."},
{"role": "user", "content": "Context: In a civil negligence action the plaintiff must prove duty, breach, causation, and damages by a preponderance of the evidence.\n\nQuestion: What standard of proof applies to the plaintiff?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt", add_special_tokens=False),
max_new_tokens=128, do_sample=False)
Limitations
- No arithmetic reliability. At 125M the model will state a revenue delta and a percentage that do not follow from the two figures it just produced. Do not trust generated numbers.
- Grounded QA only. It is trained to answer from a supplied passage. Asked open-ended questions with no context, it confabulates.
- Not legal or financial advice. Fluent, register-correct legal prose is not the same as correct legal reasoning, and this model produces the former.
- The judge that created its preference data is also the judge that evaluated it (Gemini 2.5 Flash). Win-rates against an independent judge may differ.
Training
DPO, β=0.1, effective batch 16, on 1×H100. Full pipeline, configuration, and the negative results are documented in the project repository.