slm-125m-dpo
125M-parameter legal/financial model,
preference-optimized with DPO on top of
prajwal55/slm-125m-sft.
Abstention on held-out unanswerable questions rose from 0.0% to 60.3%. It also declines on 26.3% of questions the passage does answer, and token F1 fell from 0.590 to 0.456.
This is a research artifact, not a usable model. Refusing a quarter of answerable questions is not an acceptable operating point. It is published because the measured trade-off is the result, not because the checkpoint is good.
What this phase was trying to fix
The supervised model answers questions its passage does not support — it invents a plausible fact instead of declining. That failure has no gradient in supervised fine-tuning, because the training data contains 8,000 examples of answering and none of declining.
Two axes only: abstention (decline when unsupported) and faithfulness (when supported, keep every claim traceable to a span).
Results
All checkpoints below were trained from the same SFT policy on the same 1,210 frozen preference triples, so the only variable is the algorithm and its hyperparameters. Held-out set is 121 unanswerable prompts, carved before any training and disjoint by source passage. Greedy decoding.
| Metric | SFT baseline | DPO beta=0.1, 1 epoch | DPO beta=0.1, 2 epochs | DPO beta=0.3 | DPO beta=0.5 | PPO |
|---|---|---|---|---|---|---|
| Abstention (held-out, 121) | 0.0% | 60.3% | 66.1% | 0.0% | 0.0% | 0.0% |
| False abstention | 0.0% | 26.3% | 34.7% | 0.0% | 0.0% | 0.0% |
| Token F1 (1,003 answerable) | 0.590 | 0.456 | 0.400 | 0.585 | 0.592 | 0.563 |
| Exact match | 11.4% | 6.6% | 4.8% | 11.3% | 11.8% | 11.9% |
| Numeric fidelity | 98.1% | 99.2% | 99.2% | 98.8% | 99.0% | 93.5% |
| KL from SFT | 0.0000 | 0.3159 | 0.6792 | 0.0227 | 0.0166 | 0.0628 |
Abstention alone is gameable — a model that declines everything scores 100%. Read it beside false abstention or not at all.
Why this is the 1-epoch checkpoint
The configured run was 2 epochs. Its own epoch-1 checkpoint strictly dominates the finished one:
| abstention | false abstention | ratio | token F1 | KL from SFT | |
|---|---|---|---|---|---|
| epoch 1 (this model) | 60.3% | 26.3% | 2.29 | 0.456 | 0.316 |
| epoch 2 | 66.1% | 34.7% | 1.90 | 0.400 | 0.679 |
The second epoch bought +5.8 points of abstention for +8.4 points of false abstention — a losing trade — while doubling the drift from the SFT policy. (Ratio is abstention ÷ false abstention; 1.0 means declining at random.)
The headline finding
DPO can consume off-policy chosen text; PPO cannot. With a 0.00% base
rate for the target behaviour, DPO was handed decline text and moved likelihood
mass onto it directly. PPO could only reinforce what it sampled, and it never
sampled a decline.
Abstention has a KL threshold, not a smooth trade-off. Nothing below KL ~0.06 from the SFT policy abstains at all; token F1 falls monotonically with KL regardless of which algorithm caused the drift.
Usage
from transformers import AutoTokenizer, LlamaForCausalLM
tok = AutoTokenizer.from_pretrained("prajwal55/slm-125m-dpo")
model = LlamaForCausalLM.from_pretrained("prajwal55/slm-125m-dpo")
prompt = ("<|bos|><|system|>You are a legal and financial assistant. Answer the "
"question using only the passage provided. Be precise and concise."
"<|user|>" + passage + "\n\nQuestion: " + question + "<|assistant|>")
ids = tok(prompt, add_special_tokens=False, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=160, do_sample=False,
eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
Resolve stop and pad ids from the tokenizer, not from LlamaConfig
defaults — this tokenizer is bos=0, eos=1, pad=2.
Limitations
- Not usable closed-book. It is a grounded-extraction model over a supplied passage, at 125.8M parameters.
- Declines on more than a quarter of answerable questions.
- Trained on ~1,200 preference pairs. Nothing here is a general claim about DPO or PPO — it is a claim about these algorithms meeting a target behaviour with a zero base rate.
- Checkpoint:
/data/checkpoints/dpo/beta0.1/epoch-1.