slm125mlive-qa-DPO
DPO applied to
nagbhaskar55/slm125mlive-bhaskar-finetune, a QA-SFT model, to sharpen answer
quality on grounded (passage + question) prompts.
Results
| metric | before | after |
|---|---|---|
| preference accuracy (held-out) | 0.6 | 1.0 |
| chosen-vs-rejected logprob margin | 56.0583 | 75.5719 |
Trained 2 epochs, beta 0.1, 64 steps on 260 pairs (40 held out), in 38s.
How it was trained
DPO (Direct Preference Optimization) optimises the preference objective in closed form against a frozen copy of the starting model. No reward model and no sampling are involved at training time, which is why it is fast and stable.
Preference data
600 triplets (prompt / chosen / rejected), 300 grounded and 300 closed-book. This model trained on the grounded half, because that is the distribution it was supervised-fine-tuned on.
The rejected answer in each pair carries exactly one deliberately induced flaw, drawn evenly from six modes that these models actually exhibit: a wrong figure, an unsupported claim, vagueness, rambling repetition, a partial answer, and a false refusal. An LLM judge then confirmed the ordering with the two answers shown in randomised positions, so it could not score well by always choosing the first. Pairs the judge disagreed with, or was unconfident about, were dropped.
Prompt format
This model expects slm_chat:
<|bos|><|system|>{system}<|eos|><|user|>{passage}\n\n{question}<|eos|><|assistant|>
The format was established empirically, by scoring known-good answers under competing templates and comparing generation behaviour -- not assumed from the tokenizer.
Limits
600 preference pairs is a small budget for preference optimization. Expect sharper formatting, less padding and more consistent refusals -- not new capability. Preference labels come from an LLM judge, so the model inherits that judge's blind spots. Not legal or financial advice.