sherlock-qwen2.5-3b-dpo-v1
Qwen2.5-3B-Instruct post-trained to solve MATH (levels 1-3) problems in the voice of Sherlock Holmes. CS2881R Assignment 1, week 2 (RLAIF). Full weights: the LoRA adapters are already merged in.
Lineage: Qwen/Qwen2.5-3B-Instruct -> SFT (andreayhchen/sherlock-qwen2.5-3b-sft-v1, merged) -> DPO on judge-labelled pairs
Training: TRL DPOTrainer, 619 pairs built from the scored GRPO rollouts (score gap >= 2, matched on use of "Watson" and on correctness), 2 epochs, LoRA r=16, lr 5e-5, beta 0.1.
Reward / labels: an LLM judge (claude-sonnet-5, thinking disabled) scoring persona 1-10 against a written rubric.
Responses breaking the format contract (no Final answer: \boxed{} line, outside 80-500 tokens, bullets/headers) got reward 0.
Correctness was not rewarded.
Prompt format (required): no system prompt; the user message is the problem followed by
\n\nPlease reason step by step, and put your final answer within \boxed{}.
Held-out results (200 MATH test problems, levels 1-3, greedy; persona = mean judge score):
| model | persona | accuracy |
|---|---|---|
| base Qwen2.5-3B-Instruct | - | 80% |
| SFT | 5.27 | 59% |
| GRPO, 60 steps | 5.80 | 56% |
| DPO | 7.28 | 60% |
Known issue: the judge adds about one point for the name "Watson" alone, and both RL stages increased its use.