Llama-3.2-1B-Instruct-DPO-HH
meta-llama/Llama-3.2-1B-Instruct aligned with Direct Preference Optimization on Anthropic HH-RLHF,
using a from-scratch DPO implementation — no TRL, no DPOTrainer, no Axolotl, no
Lightning. The loss, sequence scoring, completion masking, reference handling and
training loop are all explicit PyTorch.
This is research and learning code. Read the limitations before using it.
Training
| Base model | meta-llama/Llama-3.2-1B-Instruct |
| Objective | DPO (Rafailov et al., 2023), summed completion log-probabilities |
| Reference | Frozen copy of the base model, live (not cached) |
| Dataset | Anthropic/hh-rlhf, train split |
| Pairs after parsing | 159,384 |
| Pairs seen | 79,360 (50% of one epoch, 1,240 steps) |
| Hardware | 1× NVIDIA H200 |
Hyperparameters
| β | 0.1 |
| Learning rate | 5e-07 |
| Schedule | cosine to 0, 10% warmup |
| Optimizer | AdamW, β₁ 0.9, β₂ 0.95, wd 0.0 |
| Effective batch | 64 pairs (8 per device × 8 accumulation) |
| Max grad norm | 1.0 |
| Parameter dtype | float32 (BF16 autocast for compute) |
| Max length | 1024 (prompt 640, completion 384) |
| Seed | 42 |
On precision. Parameters are stored in FP32 and BF16 is used only for forward/backward compute. Storing trainable parameters in BF16 silently breaks preference tuning: at a weight of 0.01 the gap between representable BF16 values is 6.1e-5, while an AdamW update at these learning rates is ~1e-6, so updates round to a no-op while the loss curve still looks plausible. This pipeline rejects BF16 parameter storage for training outright.
Results
Training metrics only — no held-out evaluation was run. These are in-training statistics on the optimized data, not a measure of generalization.
| steps | loss | reward margin | reward accuracy |
|---|---|---|---|
| 0–248 | 0.6809 | +0.0420 | 0.548 |
| 248–496 | 0.6513 | +0.1620 | 0.596 |
| 496–744 | 0.6453 | +0.1894 | 0.631 |
| 744–992 | 0.6437 | +0.2152 | 0.618 |
| 992–1240 | 0.6435 | +0.2027 | 0.627 |
| final | 0.6455 | +0.2002 | 0.615 |
DPO's loss is exactly log 2 = 0.693147 when policy and reference are identical. This
run began at 0.690420 with implicit rewards of -0.0112 /
-0.0176, confirming the frozen reference and the completion masking were
correct at step 0.
Where the margin comes from. chosen_reward moved +0.1242 → +0.0333
and rejected_reward +0.0595 → -0.1797. The separation is
produced mainly by pushing the rejected responses below the reference, not by making the
chosen ones more likely — the likelihood-displacement behaviour DPO is known for. Read the
margin as "less likely to produce the dispreferred response", which is not the same claim
as "more likely to produce the preferred one".
How large is the effect, really?
Stated plainly, because a reward margin alone does not tell you this:
- Relative weight change from the base model: 0.000354
- Greedy generations that are byte-identical to the base model: 0 of 4 spot-check prompts
This is a modest perturbation of the base model, which is what half an epoch at lr 5e-7 with β=0.1 should produce — β exists precisely to keep the policy near its reference. On general prompts the outputs are recognisably the base model's, with differences in phrasing and formatting. Do not expect a dramatically different assistant.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "jackf857/Llama-3.2-1B-Instruct-DPO-HH"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "Explain why the sky is blue, briefly."}]
ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids.to(model.device), max_new_tokens=128)
print(tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
The Llama 3 chat template injects the current date unless date_string is pinned.
Training used date_string="26 Jul 2024".
Data processing
HH transcripts were parsed into canonical (messages, chosen, rejected) triples with no
chat markup, then rendered through the official Llama 3 chat template. Only the final
assistant response is scored; the completion mask is exactly
[0]*prompt_len + [1]*completion_len, enforced by the prompt-prefix invariant.
Verified against the real Llama 3 tokenizer: exactly one BOS per sequence, 0% BPE merges
across the prompt/completion boundary, every completion ending in <|eot_id|>.
Limitations
- No held-out evaluation. Every number above is a training metric. There is no evidence here that this model is better than its base — only that DPO optimized what it was asked to optimize.
- 50% of one epoch on a 1B model. A short run on a small model.
- HH-RLHF is noisy. Preference labels are known to be inconsistent, and many pairs have no clear quality difference. Some labels prefer epistemic humility ("I don't know") over confident answers, so the model may become more hedging.
- Safety is not established. No safety evaluation was performed. Do not deploy where harmful output matters. Inherits all limitations of the base model.
- English only; 1B models hallucinate readily.
License
Governed by the Llama 3.2 Community License, inherited from the base model. That license requires derivative model names to begin with "Llama". Anthropic HH-RLHF is MIT licensed.
Citation
@inproceedings{rafailov2023direct,
title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
author = {Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and
Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea},
booktitle = {Advances in Neural Information Processing Systems},
year = {2023}
}