Model Card for Llama-3-8B-Instruct-Iterative-SamPO
This repository provides a fine-tuned version of Llama-3-8B-Instruct, using our proposed SamPO algorithm: Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence. We obey all licenses mentioned in llama3's work.
Performance
| Model | GSM8K | IFEval | PiQA | MMLU | TruthfulQA | AlpacaEval2 | LC AlpacaEval2 | Length in Tokens |
|---|---|---|---|---|---|---|---|---|
| Llama3-8B-Instruct | 75.06 | 49.40 | 80.69 | 63.85 | 36.47 | 22.57 | 22.92 | 421 |
| Llama3-8B-Instruct-DPO | 75.59 | 51.80 | 81.94 | 64.06 | 40.39 | 23.34 | 23.20 | 422 |
| Llama3-8B-Instruct-Iterative-DPO | 74.91 | 52.52 | 81.66 | 64.02 | 39.90 | 23.92 | 25.50 | 403 |
| Llama3-8B-Instruct-Iterative-SamPO | 77.81 | 60.55 | 81.18 | 64.12 | 44.07 | 30.68 | 35.14 | 377 |
Evaluation Details
Five conditional benchmarks, using lm-evaluation-harness:
- GSM8K: 8-shot, report strict match
- IFEval: 3-shot, report instruction-level strict accuracy
- PiQA: 3-shot, report accuracy
- MMLU: 0-shot, report normalized accuracy
- TruthfulQA: 3-shot, report accuracy of single-true mc1 setting
One open-ended benchmark, using official alpaca_eval:
- AlpacaEval2: win rate (%) judged by GPT-4-turbo between the model's outputs vs. the GPT-4-turbo's response
- LC AlpacaEval2: length-debiased win rate (%) of AlpacaEval2
- Length in Tokens: the average output length of AlpacaEval2, calculated in tokens with Llama3's tokenizer
Input Format
The model is trained to use the following format:
<|start_header_id|>user<|end_header_id|>
{PROMPT}<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
{Response}
Training hyperparameters
The following hyperparameters were used during DPO/SamPO training:
- DPO beta: 0.1
- learning_rate: 4e-7
- total_train_batch_size: 128
- optimizer: AdamW with beta1 0.9, beta2 0.999 and epsilon 1e-8
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- Weight Decay: 0.0
- num_epochs: 3.0
- Specifically add above input format over training samples