UW1-DPO-soft
DPO on the mean-preference oracle with soft labels, beta_DPO = 0.1
Fine-tuned from Qwen/Qwen2.5-7B-Instruct. One of five arms trained and evaluated under a single
protocol so that the only difference between them is the loss.
Training
| Panel | WildChecklists, four native checklist items per prompt |
| Prompts | 319 train / 83 dev |
| Learner pairs | 8,932 train (all 28 unordered pairs of 8 sampled responses) |
| Optimizer steps | 140 at batch size 128, two epochs |
| Learning rate | 3e-7 cosine, warmup 0.1, weight decay 1e-6, grad clip 1.0 |
| Policy seed | 42 (one seed) |
| Candidate pool | 8 learner + 8 reference occurrences, temperature 1, top-p 1, 1024 tokens |
| Training judge | local open-weight judge, two judgments per pair and item, both orders |
Measurements
Capability columns come from one local harness at the declared few-shot counts; the two preference columns are win rates against each benchmark's released baseline answers, judged by a local 72B open-weight model in both presentation orders.
| Model | MMLU | ARC-C | HellaSwag | WinoGrande | TruthfulQA | GSM8K | IFEval | Arena-Hard | AlpacaEval | MT-Bench |
|---|---|---|---|---|---|---|---|---|---|---|
| base (untrained) | 0.7427 | 0.6724 | 0.8141 | 0.7498 | 0.6480 | 0.7210 | 0.5675 | 0.7165 | 0.3075 | 8.3063 |
| UW1-NBPO-PW | 0.7431 | 0.6706 | 0.8148 | 0.7561 | 0.6471 | 0.7240 | 0.5730 | 0.7280 | 0.3149 | 8.3125 |
| UW1-FixedRefNash-PW | 0.7431 | 0.6715 | 0.8134 | 0.7553 | 0.6474 | 0.7293 | 0.5712 | 0.7315 | 0.3180 | 8.3000 |
| UW1-DPO-soft | 0.7430 | 0.6706 | 0.8129 | 0.7561 | 0.6475 | 0.7301 | 0.5656 | 0.7100 | 0.3146 | 8.2250 |
| UW1-INPO-soft | 0.7428 | 0.6715 | 0.8144 | 0.7514 | 0.6469 | 0.7271 | 0.5823 | 0.7435 | 0.3332 | 8.2025 |
| UW1-SPPO | 0.7427 | 0.6732 | 0.8148 | 0.7514 | 0.6439 | 0.7331 | 0.5693 | 0.7430 | 0.3668 | 8.3526 |
| UW1-PROSPER | 0.7427 | 0.6698 | 0.8135 | 0.7474 | 0.6485 | 0.7301 | 0.5545 | 0.7245 | 0.3053 | 8.3019 |
Three limits that apply to every row
- The prompt set is a filtered subset. 2,000 prompts were requested; 750 were judged under a reduced budget, 660 had complete feedback, 344 passed the Nash certificate and 319 survived the representability filter. The surviving prompts are those with a strictly positive bargaining surplus, so the panel is not a random sample of the dataset.
- One seed. Differences between these arms cannot be separated from policy-seed variance, and none of the differences measured so far exceeds a single standard error on any column.
- Local judges. No hosted API was called at any point. Arena-Hard normally
judges with
gpt-4-1106, AlpacaEval 2.0 withweighted_alpaca_eval_gpt4_turboand MT-Bench with GPT-4; these numbers are comparable across the five arms and are not leaderboard scores.