slm-500m-legal-ppo

A 517.8M-parameter grounded legal/financial Q&A model, tuned with PPO against a learned reward model on AI-generated feedback (RLAIF).

Honest headline

This model performs at parity with the SFT model it was trained from.

The interval straddles 0.5. PPO here did not measurably improve the model and did not measurably harm it. If you want the better-understood model, use the SFT one.

The 72 ties are the substantive result, not a footnote. Mean KL from the SFT reference across training was 0.134 — the policy barely moved, so the judge mostly cannot distinguish the two models. A numerically healthy PPO run at this scale produces nearly the same model.

Independent absolute score (2026-08-15)

Scored by Claude Sonnet on four axes out of 10 — a different model family from the Gemini judge behind the win-rate above, and one that had no hand in writing this project's training data. 300 held-out prompts, the same prompts and the same scale used for all twelve checkpoints, so this number is comparable across models in a way no win-rate here is.

The full twelve-checkpoint table, with every stage-to-stage interval, is in MODEL_INDEX.md in the project repository.

Where it sits in the scaling picture

Same pipeline, same prompt pool, same Gemini 2.5 Flash judge, same PPO code across all three models. PPO's win-rate falls with scale — the mirror image of DPO's rise:

Only the two ends are significant, and they point in opposite directions. The Gemma PPO checkpoint is a real degradation and is deliberately not published.

Evaluation caveat

The reward model distilled Gemini 2.5 Flash's preferences, PPO optimized against that reward model, and Gemini 2.5 Flash then judged the result. Win-rates under those conditions partly measure agreement with the judge's own taste. An independent judge would be needed to break that circularity.

Training

PPO maximizing reward - kl_coef * KL(policy ‖ SFT), 60 rollout steps, 16 prompts per rollout, kl_coef=0.2 on 1×H100. Realized: mean KL 0.134, final KL 0.556, value loss 0.0005–0.035, kl_coef never left its floor, no abort, 13.3 min, $0.23.

Three implementation details were load-bearing — without them the run is not merely worse, it is invalid:

• Reward whitening. kl_coef is only meaningful relative to reward magnitude, and a Bradley-Terry reward model only fixes score differences — its absolute scale is arbitrary. Scores are normalized per rollout batch.
• Gradient accumulation across the rollout, rather than one optimizer step per sequence.
• An adaptive KL controller with a floor at the configured kl_coef, since at step 1 the policy is the reference and KL ≈ 0.

Reward model

Trained from the same SFT backbone with a fresh scalar head and the pairwise Bradley-Terry loss, reaching 0.820 pairwise accuracy on held-out preference pairs. Published separately as slm-500m-legal-rm.

Prompt format

Use the chat template shipped with the tokenizer. The assistant turn ends on <|eos|>.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("abhishekai/slm-500m-legal-ppo")
model = AutoModelForCausalLM.from_pretrained("abhishekai/slm-500m-legal-ppo")

msgs = [
    {"role": "system", "content": "You are a precise legal and financial assistant. Answer only from the provided context."},
    {"role": "user", "content": "Context: In a civil negligence action the plaintiff must prove duty, breach, causation, and damages by a preponderance of the evidence.\n\nQuestion: What standard of proof applies to the plaintiff?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt", add_special_tokens=False),
                     max_new_tokens=128, do_sample=False)

Limitations

• Grounded QA only — it answers from a supplied passage and confabulates without one.
• Arithmetic is unreliable. Do not trust generated figures.
• Not legal or financial advice.
• PPO optimizes a proxy for quality; a higher reward-model score is not evidence of a more correct answer.