LLaMA3-iterative-DPO-final-GGUF

This is quantized version of RLHFlow/LLaMA3-iterative-DPO-final created using llama.cpp

Model Description

We release an unofficial checkpoint of a state-of-the-art instruct model of its class, LLaMA3-iterative-DPO-final. On all three widely-used instruct model benchmarks: Alpaca-Eval-V2, MT-Bench, Chat-Arena-Hard, our model outperforms all models of similar size (e.g., LLaMA-3-8B-it), most large open-sourced models (e.g., Mixtral-8x7B-it), and strong proprietary models (e.g., GPT-3.5-turbo-0613). The model is trained with open-sourced datasets without any additional human-/GPT4-labeling.

Even better, we provide a detailed recipe to reproduce the model. Enjoy!

Model Releases

See the collection of the training set, reward/preference model, SFT model.

• SFT model
• Reward model

Dataset

• Preference data mix
• Prompt collection for RLHF training

Training methods

We have developed a simple and efficient online RLHF recipe for LLM instruct training. Our recipe is DPO-based and thus much cheaper and simpler to train and tune compared to PPO-based approaches. Unlike widely-used offline DPO, the online component of our approach effectively mitigates distribution shifts during policy optimization. For a detailed exposition, please refer to our accompanying technical report.

Chat Benchmarks

Academic Benchmarks

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" 

model = AutoModelForCausalLM.from_pretrained("RLHFlow/LLaMA3-iterative-DPO-final")
tokenizer = AutoTokenizer.from_pretrained("RLHFlow/LLaMA3-iterative-DPO-final")

messages = [
    {"role": "user", "content": "I'm trying to teach myself to have nicer handwriting. Can you help?"},
]

model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = model_inputs.to(device)
model.to(device)

output_tokens = model.generate(model_inputs, max_new_tokens=1024, do_sample=True)
model_outputs = tokenizer.batch_decode(output_tokens)
print(model_outputs[0])

Limitations

RLHFlow/LLaMA3-iterative-DPO-final is an unofficial checkpoint developed to illustrate the power of online iterative RLHF and is for research purpose. While safety and ethical considerations are integral to our alignment process, there remains the possibility that the model could generate offensive or unethical content, particularly under adversarial conditions. We are committed to continuous improvement in our models to minimize such risks and encourage responsible usage.