sherlock-qwen2.5-3b-grpo-v1
Qwen2.5-3B-Instruct post-trained to solve MATH (levels 1-3) problems in the voice of Sherlock Holmes. CS2881R Assignment 1, week 2 (RLAIF). Full weights: the LoRA adapters are already merged in.
Lineage: Qwen/Qwen2.5-3B-Instruct -> SFT (andreayhchen/sherlock-qwen2.5-3b-sft-v1, merged) -> GRPO with the judge as the reward
Training: TRL GRPOTrainer, 60 steps x 8 prompts x 8 rollouts, LoRA r=16, lr 2e-5, KL beta 0.04 to the SFT model.
Reward / labels: an LLM judge (claude-sonnet-5, thinking disabled) scoring persona 1-10 against a written rubric.
Responses breaking the format contract (no Final answer: \boxed{} line, outside 80-500 tokens, bullets/headers) got reward 0.
Correctness was not rewarded.
Prompt format (required): no system prompt; the user message is the problem followed by
\n\nPlease reason step by step, and put your final answer within \boxed{}.
Held-out results (200 MATH test problems, levels 1-3, greedy; persona = mean judge score):
| model | persona | accuracy |
|---|---|---|
| base Qwen2.5-3B-Instruct | - | 80% |
| SFT | 5.27 | 59% |
| GRPO, 60 steps | 5.80 | 56% |
| DPO | 7.28 | 60% |
Known issue: the judge adds about one point for the name "Watson" alone, and both RL stages increased its use.