Phi-2-ORPO
Phi-2-ORPO is a fine-tuned version of microsoft/phi-2 on argilla/dpo-mix-7k preference dataset using Odds Ratio Preference Optimization (ORPO). The model has been trained for 1 epoch.
LazyORPO
This model has been trained using LazyORPO. A colab notebook that makes the training process much easier. Based on ORPO paper. This notebook has been created by Zain Ul Abideen
What is ORPO?
Odds Ratio Preference Optimization (ORPO) proposes a new method to train LLMs by combining SFT and Alignment into a new objective (loss function), achieving state of the art results. Some highlights of this techniques are:
- 🧠 Reference model-free → memory friendly
- 🔄 Replaces SFT+DPO/PPO with 1 single method (ORPO)
- 🏆 ORPO Outperforms SFT, SFT+DPO on PHI-2, Llama 2, and Mistral
- 📊 Mistral ORPO achieves 12.20% on AlpacaEval2.0, 66.19% on IFEval, and 7.32 on MT-Bench out Hugging Face Zephyr Beta
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
torch.set_default_device("cuda")
model = AutoModelForCausalLM.from_pretrained("abideen/phi2-pro", torch_dtype="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("abideen/phi2-pro", trust_remote_code=True)
inputs = tokenizer('''
"""
Write a detailed analogy between mathematics and a lighthouse.
"""''', return_tensors="pt", return_attention_mask=False)
outputs = model.generate(**inputs, max_length=200)
text = tokenizer.batch_decode(outputs)[0]
print(text)