Mistral-7B ARC-Challenge SFT + Synthetic-Pair DPO
Mistral-7B-v0.1 fine-tuned for the ARC-Challenge benchmark (Upstage AIRE
deep-learning coding test). acc_norm 71.84 ± 0.48 under the official
protocol (lm-evaluation-harness arc_challenge, 25-shot, four few-shot seeds),
vs. 61.07 ± 0.36 for the base model measured identically. The task's reference
score to beat was 61.43.
Recipe
Two stages, both LoRA (r=32, alpha=64, all seven projections), merged into bf16 for this checkpoint:
- SFT on ARC-Challenge + ARC-Easy train splits (3,370 items), rendered
byte-for-byte in the harness's own surface form (
Question: ...\nAnswer:), loss masked to the answer span. lr 1e-4 cosine, 2 epochs. - DPO on 8,025 gold-vs-distractor pairs drawn from a synthetic pool: 2,675 teacher-generated (Qwen3.8-27B), label-verified, ARC-form-matched MCQs that the SFT model still failed ("hard mining"). beta 0.1, with hard-pair gating and a chosen-NLL anchor.
The synthetic pool and its generation/verification pipeline are published separately (see the companion dataset repo), including the contamination audit: generated items were screened against ARC-C/E test+validation by exact and 8-gram overlap; the teacher was only ever shown ARC train items.
Evaluation
lm_eval --model hf --model_args pretrained=<this repo>,dtype=float16 \
--tasks arc_challenge --num_fewshot 25 --seed 0,1234,1234,<fewshot_seed>
| model | acc_norm (4-seed mean ± sd) |
|---|---|
| Mistral-7B-v0.1 (base) | 61.07 ± 0.36 |
| + SFT only | 67.04 ± 0.50 |
| + SFT + DPO (this) | 71.84 ± 0.48 |
| Qwen3.8-27B (teacher) | 73.21 (single seed) |
Trained on 2x RTX A6000; the SFT stage also reproduces on a single T4 in its QLoRA variant (67.66 ± 0.36, statistically equivalent to bf16 SFT).