ORPO fine-tune of Mistral 7B v0.1 with DPO Mix 7K
Stable Diffusion XL "A capybara, a killer whale, and a robot named Ultra being friends"
This is an ORPO fine-tune of mistralai/Mistral-7B-v0.1 with
alvarobartt/dpo-mix-7k-simplified.
⚠️ Note that the code is still experimental, as the ORPOTrainer PR is still not merged, follow its progress
at 🤗trl - ORPOTrainer PR.
About the fine-tuning
In order to fine-tune mistralai/Mistral-7B-v0.1 using ORPO, the branch
orpo from 🤗trl has been used, thanks to the invaluable and quick contribution of
@kashif.
ORPO stands for Odds Ratio Preference Optimization, and defines a new paradigm on fine-tuning LLMs, “combining” both the SFT and the PPO/DPO stage into a single stage, thanks to the proposed loss function starting off from a preference dataset i.e. chosen-rejected pairs.
Some key features about ORPO:
- ⚡️ Faster to train as it’s now a single stage fine-tuning
- 👨🏻🏫 Requires preference data i.e. (prompt, chosen, rejected)-like datasets
- ⬇️ Less memory than PPO/DPO as doesn’t need a reference model
- 🏆 SOTA results for Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) when fine-tuned using single-turn UltraFeedback
Some notes on the experiments mentioned in the paper:
- 📌 Up to 7B parameter LLMs were fine-tuned, achieving better performance compared to 7B counterparts and even 13B LLMs
- 📌 Not yet trained with multi-turn datasets as Capybara (may be an interesting experiment to run)
- 📌 For OPT models fine-tuned with HH-RLHF from Anthropic, truncated and padded to 1024 tokens, filtering out filtering the prompts with > 1024 tokens
- 📌 For Phi-2, Mistral (7B) and Llama 2 (7B), or UltraFeedback from OpenBMB (truncated and padded to 2048 tokens), filtering out filtering the prompts with > 1024 tokens
- 📌 Fine-tuned for 10 epochs, and using the evaluation loss as the metric for selecting the best models
For more information about ORPO, I highly recommend reading their paper titled ORPO: Monolithic Preference Optimization without Reference Model,
as it contains a lot of information and details not only on the ORPO method, but also on the experiment they ran, the results they got, and much more.
📅 Fine-tuning code will be shared soon, stay tuned!
About the dataset
The dataset used for this fine-tune is alvarobartt/dpo-mix-7k-simplified,
which is a simplified version of argilla/dpo-mix-7k.
The simplification comes from the fact that the prompt column is detached from both the chosen and rejected
columns so that there's no need for extra pre-processing while applying the chat template to the dataset before the
fine-tuning. So on, the dataset remains as is, with an additional column for the prompt.
The dataset is a small cocktail combining Argilla's latest efforts on DPO datasets, mixing the following datasets:
argilla/distilabel-capybara-dpo-7k-binarizedargilla/distilabel-intel-orca-dpo-pairsargilla/ultrafeedback-binarized-preferences-cleaned
The samples have been randomly selected from the original datasets with a proportion of 0.33 each, as can be seen via the dataset column of the dataset.
For more information about the original dataset check the README.md file of argilla/dpo-mix-7k.