Qwen3-4B-DPO-prm-pairs
DPO fine-tune of Qwen/Qwen3-4B-Instruct-2507 on preference pairs of supervisor feedback for an LLM coding agent. The task the model is trained toward: given an agent's trajectory, produce trajectory-level error analysis and corrective guidance, preferring higher-quality critiques.
Trained with TRL's DPOTrainer.
Results — read before using
The run improved over its own baseline but did not exceed chance on pairwise preference accuracy. Reported honestly:
| epoch | eval_loss | eval_rewards/accuracies | eval_rewards/margins |
|---|---|---|---|
| 0.00 (baseline) | 0.6914 | 0.0000 * | 0.0000 |
| 0.23 | 0.7172 | 0.3187 | −0.0073 |
| 0.46 | 0.6926 | 0.3875 | +0.0746 |
| 0.69 | 0.6925 | 0.4562 | +0.0866 |
| 0.92 | 0.6949 | 0.4313 | +0.0877 |
| 1.00 (final) | 0.6680 | 0.4375 | +0.1310 |
* At step 0 the policy is the reference model, so both rewards are exactly 0
and the strict chosen > rejected comparison is False for every pair. That 0 is
a degenerate tie, not a meaningful score.
Final train_loss 0.666.
Interpretation. Eval loss ended below baseline (0.668 vs 0.6914) and the
reward margin grew steadily to +0.131, so the model does separate some pairs.
But rewards/accuracies finished at 0.4375, below the 0.5 chance line — a
positive mean margin combined with a sub-chance win rate means the average is
carried by a minority of strongly-separated pairs while most pairs remain
slightly mis-ranked.
The metric curve was still climbing at the end of epoch 1, so this looks
under-trained rather than converged: only 44 optimizer steps were taken at
learning_rate=5e-7. Treat this as a checkpoint of record, not a model that
reliably beats its base at preference ranking.
Training details
- Base model: Qwen/Qwen3-4B-Instruct-2507 (4.02B params)
- Method: DPO (
trl.DPOTrainer), β = 0.1 (default) - Data: 1,541 preference pairs (ShareGPT-style
messages+chosen/rejected), split 90/10 → 1,386 train / 155 eval (seed 42). Pairs are pre-filtered to fit whole within 8,192 tokens; median length ≈ 7.7k tokens. - max_length: 8192 (nothing truncated)
- Precision: bf16 (mixed precision via Accelerate)
- Hardware: 8 × NVIDIA L40S (46 GB), single node
- Parallelism: DeepSpeed ZeRO-3, no offload, no gradient checkpointing (peak ≈ 41.5 GB/GPU)
- Batching:
per_device_train_batch_size=1×gradient_accumulation_steps=4× 8 GPUs = effective batch 32 → 44 optimizer steps for 1 epoch - learning_rate: 5e-7
- epochs: 1
- Runtime: 1 h 09 m (≈ 79 s/step; evals every 10 steps at ≈ 158 s each)
Reproducing
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch \
--config_file examples/accelerate_configs/deepspeed_zero3.yaml \
dpo_prm_pairs.py
Known limitations
- Sub-chance pairwise preference accuracy (see above) — do not assume it out-ranks the base model on this task.
- Under-trained at 44 optimizer steps; a larger step count (e.g. halving
gradient_accumulation_stepsto restore ~87 steps) or a higher learning rate is the obvious next experiment. - Single epoch, single seed, no hyperparameter search.
- Evaluated only on a 155-pair in-distribution holdout. No general-capability benchmarks were run, so regression on base-model abilities is unmeasured.
- Trained on long-context (≈7.7k token) agent traces; behavior on short prompts is untested.