🔥 Chat with Magpie Here!
🐦 Llama-3.1-8B-Magpie-Align-v0.1
Project Web: https://magpie-align.github.io/
Online Model Demo: https://huggingface.co/spaces/flydust/Chat-with-Magpie
Arxiv Technical Report: https://arxiv.org/abs/2406.08464
Codes: https://github.com/magpie-align/magpie
🧐 About This Model
This model is an aligned version of meta-llama/Meta-Llama-3.1-8B. We apply the following pipeline:
We first perform SFT using:
- Magpie-Align/Magpie-Pro-MT-300K-v0.1, and
- Magpie-Align/Magpie-Reasoning-150K
- SFT Model Checkpoint: Magpie-Align/Llama-3.1-8B-Magpie-Align-SFT-v0.1
We then perform DPO on the princeton-nlp/llama3-ultrafeedback-armorm dataset.
The overall performance is much better than the official Llama-3.1-8B-Instruct Model!
- MT-Bench: 8.375 (1st Turn), 7.650 (Second Turn), 8.013 (Average)
- Alpaca Eval 2 (GPT-4-Turbo-1106): 45.73 (LC), 52.79 (WR)
- Arena Hard: 42.4
👀 Other Information
License: Please follow Meta Llama 3 Community License (Data) and Meta Llama 3.1 Community License (Model).
Conversation Template: Please use Llama 3 official chat template for the best performance.
How to use it? Please check the official Llama 3.1 repository for detailed instructions. Simply replace the original model_id with this model id.
Alignment Pipeline
The detailed alignment pipeline is as follows.
Stage 1: Supervised Fine-tuning
We use Axolotl for SFT. Please refer to the model card of SFT checkpoint for detailed configurations.
Stage 2: Direct Preference Optimization
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-06
- train_batch_size: 2
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 16
- total_train_batch_size: 256
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.4439 | 0.4275 | 100 | 0.4168 | -4.9964 | -6.3086 | 0.8145 | 1.3123 | -829.7151 | -704.6570 | -0.5150 | -0.5001 |
| 0.343 | 0.8549 | 200 | 0.3298 | -4.9310 | -6.7966 | 0.8952 | 1.8655 | -878.5105 | -698.1248 | -0.5776 | -0.5622 |
Framework versions
- Transformers 4.43.2
- Pytorch 2.3.1+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1
# Customized Configs
model_name_or_path: Magpie-Align/Llama-3.1-8B-Magpie-Align-SFT-v0.1
hub_model_id: Magpie-Align/Llama-3.1-8B-Magpie-Align-v0.1
output_dir: /data/zhangchen_xu/alignment_handbook_out/Llama-3.1-8B-Magpie-Align-v0.1
run_name: Llama-3.1-8B-Magpie-Align-v0.1
dataset_mixer:
princeton-nlp/llama3-ultrafeedback-armorm: 1.0
dataset_splits:
- train
- test
preprocessing_num_workers: 64
# DPOTrainer arguments
bf16: true
beta: 0.01
learning_rate: 1.0e-6
gradient_accumulation_steps: 16
per_device_train_batch_size: 2
per_device_eval_batch_size: 4
num_train_epochs: 1
max_length: 2048
max_prompt_length: 1800
warmup_ratio: 0.1
logging_steps: 1
lr_scheduler_type: cosine
optim: adamw_torch
torch_dtype: null
use_flash_attention_2: true
do_eval: true
evaluation_strategy: steps
eval_steps: 100
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: False
log_level: info
push_to_hub: true
save_strategy: "steps"
save_steps: 100
save_total_limit: 1
seed: 42
report_to:
- wandb
Paper Abstract
📚 Citation
If you find the model, data, or code useful, please cite our paper:
@article{xu2024magpie,
title={Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing},
author={Zhangchen Xu and Fengqing Jiang and Luyao Niu and Yuntian Deng and Radha Poovendran and Yejin Choi and Bill Yuchen Lin},
year={2024},
eprint={2406.08464},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
Please also cite the creators of preference datasets:
SimPO paper:
@article{meng2024simpo,
title={{SimPO}: Simple preference optimization with a reference-free reward},
author={Meng, Yu and Xia, Mengzhou and Chen, Danqi},
journal={arXiv preprint arXiv:2405.14734},
year={2024}
}
UltraFeedback paper:
@article{cui2023ultrafeedback,
title={{UltraFeedback}: Boosting language models with high-quality feedback},
author={Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and Zhu, Wei and Ni, Yuan and Xie, Guotong and Liu, Zhiyuan and Sun, Maosong},
journal={arXiv preprint arXiv:2310.01377},
year={2023}
}
ArmoRM paper:
@article{wang2024interpretable,
title={Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts},
author={Wang, Haoxiang and Xiong, Wei and Xie, Tengyang and Zhao, Han and Zhang, Tong},
journal={arXiv preprint arXiv:2406.12845},
year={2024}
}
Questions? Please contact Zhangchen by email.