🔥 Chat with Magpie Here!

🐦 Llama-3-8B-Magpie-Align-v0.3

Project Web: https://magpie-align.github.io/

Online Model Demo: https://huggingface.co/spaces/flydust/Chat-with-Magpie

Arxiv Technical Report: https://arxiv.org/abs/2406.08464

Codes: https://github.com/magpie-align/magpie

🧐 About This Model

This model is an aligned version of meta-llama/Meta-Llama-3-8B. We apply the following pipeline:

We first perform SFT using:

• Magpie-Align/Magpie-Pro-MT-300K-v0.1
• Magpie-Align/Magpie-Reasoning-150K
• Magpie-Align/Magpie-Qwen2-Pro-200K-Chinese
• SFT Model Checkpoint: Magpie-Align/Llama-3-8B-Magpie-Align-SFT-v0.3

We then perform DPO on the princeton-nlp/llama3-ultrafeedback-armorm dataset.

The overall performance is much better than the official Llama-3-8B-Instruct Model! Plus, it can answer Chinese queries frequently, thanks to our new Chinese instruction dataset!

• Alpaca Eval 2 (vs GPT-4-Turbo-1106): 48.58 (LC), 50.36 (WR)
• Alpaca Eval 2 (vs Llama-3-8B-Instruct): 73.65 (LC), 75.81 (WR)
• Arena Hard: 42.2
• WildBench WB-Score: 41.1
• Zero-Eval GSM: 50.0

🔥 Model Performance

We compare our Llama-3-8B-Magpie-Align with official and other open-aligned LLMs that have been fine-tuned from base models and have publicly released their training datasets. The results are as follows:

+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
|               Aligned Model ID              | MT-Bench |      |      |   Alpaca Eval 2    |       |     Alpaca Eval 2     |       | Arena Hard |
|                                             |          |      |      | (GPT-4-Turbo-1106) |       | (Llama-3-8B-Instruct) |       |            |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
|                                             |    R1    |  R2  |  AVG |        LC WR       |   WR  |         LC WR         |   WR  |    Score   |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
| meta-llama/Meta-Llama-3-8B-Instruct         |   8.31   | 7.65 | 7.98 |        22.92       | 22.57 |           50          |   50  |    20.6    |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
| princeton-nlp/Llama-3-Base-8B-SFT-DPO       |   8.12   | 7.23 | 7.67 |        17.71       | 15.34 |         43.73         | 38.80 |    14.8    |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
| NousResearch/Hermes-2-Pro-Llama-3-8B        |   8.05   | 7.35 | 7.70 |        15.60       | 12.86 |         36.37         | 30.52 |    11.5    |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
| allenai/llama-3-tulu-2-dpo-8b               |   7.71   | 7.15 | 7.43 |        14.89       |  14.8 |         35.43         | 35.42 |    11.7    |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
| cognitivecomputations/dolphin-2.9-llama3-8b |   7.97   | 6.98 | 7.47 |        12.50       |  8.79 |         32.67         | 22.80 |     8.2    |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
| openchat/openchat-3.6-8b-20240522           |   7.83   | 7.23 | 7.53 |        17.70       | 12.53 |         41.30         | 30.79 |     6.7    |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
| Magpie-Align/Llama-3-8B-Magpie-Align-v0.1   |   8.01   | 7.63 | 7.82 |        38.52       | 38.47 |         69.37         | 70.05 |    32.4    |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
| Magpie-Align/Llama-3-8B-Magpie-Align-v0.2   |   7.81   | 7.64 | 7.73 |        49.86       | 51.98 |         75.17         | 78.20 |    37.5    |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+
| Magpie-Align/Llama-3-8B-Magpie-Align-v0.3   |   7.82   | 7.51 | 7.67 |        48.58       | 50.36 |         73.65         | 75.81 |    42.2    |
+---------------------------------------------+----------+------+------+--------------------+-------+-----------------------+-------+------------+

👀 Other Information

License: Please follow Meta Llama 3 Community License.

Conversation Template: Please use Llama 3 official chat template for the best performance.

How to use it? Please check the official Llama 3 repository for detailed instructions. Simply replace the original model_id with Magpie-Align/Llama-3-8B-Magpie-Align-SFT-v1.0.

The detailed training pipeline is as follows.

Stage 1: Supervised Fine-tuning

We use Axolotl for SFT.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• total_eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 98
• num_epochs: 2

Training results

Framework versions

• Transformers 4.42.3
• Pytorch 2.3.1+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1

Internal name for identification: Llama-3-8B-Magpie-Mix-RC. Please change the model name in the below Axolotl config.

axolotl version: 0.4.1

base_model: meta-llama/Meta-Llama-3-8B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: Magpie-Align/Magpie-Reasoning-150K
    type: sharegpt
    conversation: llama3
  - path: Magpie-Align/Magpie-Qwen2-Pro-200K-Chinese
    type: sharegpt
    conversation: llama3
  - path: Magpie-Align/Magpie-Pro-MT-300K-v0.1
    type: sharegpt
    conversation: llama3
dataset_prepared_path: last_run_prepared
val_set_size: 0.001
output_dir: axolotl_out/Llama-3-8B-Magpie-Mix-RC

sequence_len: 8192
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

wandb_project: SynDa
wandb_entity:
wandb_watch:
wandb_name: Llama-3-8B-Magpie-Mix-RC
wandb_log_model:
hub_model_id: Magpie-Align/Llama-3-8B-Magpie-Mix-RC

gradient_accumulation_steps: 32
micro_batch_size: 1
num_epochs: 2
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_ratio: 0.1
evals_per_epoch: 5
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  pad_token: <|end_of_text|>

Stage 2: Direct Preference Optimization

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 7e-07
• train_batch_size: 2
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 8
• total_train_batch_size: 128
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Framework versions

• Transformers 4.42.3
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1

# Customized Configs
model_name_or_path: Magpie-Align/Llama-3-8B-Magpie-Align-SFT-v0.3
hub_model_id: Magpie-Align/Llama-3-8B-Magpie-Align-v0.3-RC
output_dir: alignment_handbook_out/Llama-3-8B-Magpie-Align-v0.3-RC
run_name: Llama-3-8B-Magpie-Align-v0.3-RC

dataset_mixer:
  princeton-nlp/llama3-ultrafeedback-armorm: 1.0
dataset_splits:
- train
- test
preprocessing_num_workers: 24

# DPOTrainer arguments
bf16: true
beta: 0.01
learning_rate: 0.7e-6
gradient_accumulation_steps: 8
per_device_train_batch_size: 2
per_device_eval_batch_size: 4
num_train_epochs: 1
max_length: 2048
max_prompt_length: 1800
warmup_ratio: 0.1
logging_steps: 1
lr_scheduler_type: cosine
optim: adamw_torch

torch_dtype: null
use_flash_attention_2: true
do_eval: true
evaluation_strategy: steps
eval_steps: 100
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: False
log_level: info
push_to_hub: true
save_strategy: "steps"
save_steps: 100
save_total_limit: 1
seed: 42
report_to:
- wandb

Downstream Performance (Lighteval)

Paper Abstract

📚 Citation

If you find the model, data, or code useful, please cite our paper:

@article{xu2024magpie,
	title={Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing}, 
	author={Zhangchen Xu and Fengqing Jiang and Luyao Niu and Yuntian Deng and Radha Poovendran and Yejin Choi and Bill Yuchen Lin},
	year={2024},
	eprint={2406.08464},
	archivePrefix={arXiv},
	primaryClass={cs.CL}
}

Please also cite the creators of preference datasets:

SimPO paper:

@article{meng2024simpo,
  title={{SimPO}: Simple preference optimization with a reference-free reward},
  author={Meng, Yu and Xia, Mengzhou and Chen, Danqi},
  journal={arXiv preprint arXiv:2405.14734},
  year={2024}
}

UltraFeedback paper:

@article{cui2023ultrafeedback,
  title={{UltraFeedback}: Boosting language models with high-quality feedback},
  author={Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and Zhu, Wei and Ni, Yuan and Xie, Guotong and Liu, Zhiyuan and Sun, Maosong},
  journal={arXiv preprint arXiv:2310.01377},
  year={2023}
}

ArmoRM paper:

@article{wang2024interpretable,
  title={Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts},
  author={Wang, Haoxiang and Xiong, Wei and Xie, Tengyang and Zhao, Han and Zhang, Tong},
  journal={arXiv preprint arXiv:2406.12845},
  year={2024}
}

Questions? Please contact Zhangchen by email.