🔥 Chat with Magpie Here!

🐦 Llama-3-8B-Magpie-Align-v0.1

Project Web: https://magpie-align.github.io/

Online Model Demo: https://huggingface.co/spaces/flydust/Chat-with-Magpie

Arxiv Technical Report: https://arxiv.org/abs/2406.08464

Codes: https://github.com/magpie-align/magpie

Model Overview

This model is an aligned version of meta-llama/Meta-Llama-3-8B. We apply the following pipeline:

• We first use Magpie-Align/Magpie-Pro-MT-300K-v0.1 dataset and perform SFT -> Magpie-Align/Llama-3-8B-Magpie-Align-SFT-v0.1
• We then perform DPO on the princeton-nlp/llama3-ultrafeedback dataset.

The overall performance is even better than the official Llama-3-8B-Instruct Model!

• Alpaca Eval 2 (vs GPT-4-Turbo-1106): 38.52 (LC), 38.47 (WR)
• Alpaca Eval 2 (vs Llama-3-8B-Instruct): 69.37 (LC), 70.05 (WR)
• Arena Hard: 32.4
• WildBench: 39.3 ((was) Best <30B Model! 🏆)
• Zero-Eval GSM: 54.62

Model Performance

We compare our Llama-3-8B-Magpie-Align with official and other open-aligned LLMs that have been fine-tuned from base models and have publicly released their training datasets. The results are as follows:

+---------------------------------------------+--------------------+--------------------+-----------------------+------------+
|               Aligned Model ID              |      MT-Bench      |   Alpaca Eval 2    |     Alpaca Eval 2     | Arena Hard |
|                                             |                    | (GPT-4-Turbo-1106) | (Llama-3-8B-Instruct) |            |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
|                                             |  R1  |  R2  |  AVG |   LC WR  |    WR   |   LC WR   |     WR    |    Score   |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| meta-llama/Meta-Llama-3-8B-Instruct         | 8.31 | 7.65 | 7.98 |   22.92  |  22.57  |     50    |     50    |    20.6    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| princeton-nlp/Llama-3-Base-8B-SFT-DPO       | 8.12 | 7.23 | 7.67 |   17.71  |  15.34  |   43.73   |   38.80   |    14.8    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| NousResearch/Hermes-2-Pro-Llama-3-8B        | 8.05 | 7.35 | 7.70 |   15.60  |  12.86  |   36.37   |   30.52   |    11.5    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| allenai/llama-3-tulu-2-dpo-8b               | 7.71 | 7.15 | 7.43 |   14.89  |  14.80  |   35.43   |   35.42   |    11.7    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| cognitivecomputations/dolphin-2.9-llama3-8b | 7.97 | 6.98 | 7.47 |   12.50  |   8.79  |   32.67   |   22.80   |     8.2    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| openchat/openchat-3.6-8b-20240522           | 7.83 | 7.23 | 7.53 |   17.70  |  12.53  |   41.30   |   30.79   |     6.7    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| Magpie-Align/Llama-3-8B-Magpie-Align-v0.1   | 8.01 | 7.63 | 7.82 |   38.52  |  38.47  |   69.37   |   70.05   |    32.4    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| Magpie-Align/Llama-3-8B-Magpie-Align-v0.2   | 7.81 | 7.64 | 7.73 |   49.86  |  51.98  |   75.17   |   78.20   |    37.5    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+

👀 Other Information

License: Please follow Meta Llama 3 Community License.

Conversation Template: Please use Llama 3 official chat template for the best performance.

How to use it? Please check the official Llama 3 repository for detailed instructions. Simply replace the original model_id with Magpie-Align/Llama-3-8B-Magpie-Align-v0.1.

The detailed training pipeline is as follows.

Stage 1: Supervised Fine-tuning

We use Axolotl for SFT.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 2

Training results

Framework versions

• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1

axolotl version: 0.4.0

base_model: meta-llama/Meta-Llama-3-8B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: Magpie-Align/Magpie-Pro-MT-300K-v0.1
    type: sharegpt
    conversation: llama3
dataset_prepared_path: last_run_prepared
val_set_size: 0.001
output_dir: ./out_Llama-3-8B-Magpie-Pro-300K-MT

sequence_len: 8192
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

gradient_accumulation_steps: 8
micro_batch_size: 1
num_epochs: 2
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
evals_per_epoch: 3
eval_table_size:
saves_per_epoch: 3
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  pad_token: <|end_of_text|>

Stage 2: Direct Preference Optimization

We use alignment handbook for DPO.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 2
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

It achieves the following results on the evaluation set:

• Loss: 0.6084
• Rewards/chosen: -1.6265
• Rewards/rejected: -1.9735
• Rewards/accuracies: 0.6809
• Rewards/margins: 0.3470
• Logps/rejected: -458.6070
• Logps/chosen: -418.2021
• Logits/rejected: -0.6447
• Logits/chosen: -0.6439

Framework versions

• Transformers 4.41.2
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1

# Model arguments
model_name_or_path: Magpie-Align/Llama-3-8B-Magpie-Pro-MT-SFT-v0.1
torch_dtype: null

# Data training arguments
# For definitions, see: src/h4/training/config.py
dataset_mixer:
  princeton-nlp/llama3-ultrafeedback: 1.0
dataset_splits:
- train
- test
preprocessing_num_workers: 12

# DPOTrainer arguments
bf16: true
beta: 0.01
do_eval: true
evaluation_strategy: steps
eval_steps: 100
gradient_accumulation_steps: 16
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: False
hub_model_id: Magpie-Align/Llama-3-8B-Magpie-Pro-MT-UltraDPO2
learning_rate: 1.0e-6
log_level: info
logging_steps: 1
lr_scheduler_type: cosine
max_length: 2048
max_prompt_length: 1800
num_train_epochs: 1
optim: adamw_torch
output_dir: data/magpie-pro-mt-ultradpo-1e-6
per_device_train_batch_size: 2
per_device_eval_batch_size: 4
push_to_hub: true
save_strategy: "steps"
save_steps: 100
save_total_limit: 1
seed: 42
warmup_ratio: 0.1

Downstream Performance

Paper Abstract

📚 Citation

If you find the model, data, or code useful, please cite our paper:

@article{xu2024magpie,
	title={Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing}, 
	author={Zhangchen Xu and Fengqing Jiang and Luyao Niu and Yuntian Deng and Radha Poovendran and Yejin Choi and Bill Yuchen Lin},
	year={2024},
	eprint={2406.08464},
	archivePrefix={arXiv},
	primaryClass={cs.CL}
}

Please also cite the creators of preference datasets:

SimPO paper:

@article{meng2024simpo,
  title={{SimPO}: Simple preference optimization with a reference-free reward},
  author={Meng, Yu and Xia, Mengzhou and Chen, Danqi},
  journal={arXiv preprint arXiv:2405.14734},
  year={2024}
}

UltraFeedback paper:

@article{cui2023ultrafeedback,
  title={{UltraFeedback}: Boosting language models with high-quality feedback},
  author={Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and Zhu, Wei and Ni, Yuan and Xie, Guotong and Liu, Zhiyuan and Sun, Maosong},
  journal={arXiv preprint arXiv:2310.01377},
  year={2023}
}

ArmoRM paper:

@article{wang2024interpretable,
  title={Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts},
  author={Wang, Haoxiang and Xiong, Wei and Xie, Tengyang and Zhao, Han and Zhang, Tong},
  journal={arXiv preprint arXiv:2406.12845},
  year={2024}
}

Questions? Please contact Zhangchen by email.