Gemma 2 9B Deita SFT Epoch 3 DPO (Epoch 2)
This repository contains the epoch-2 inference export of a DPO fine-tune of
iamhyunmin/gemma2-9b-deita-sft-epoch3.
The base model is derived from Google Gemma 2 9B.
Modification notice
The model weight files in this repository were modified from the base model by full-parameter Direct Preference Optimization (DPO). This is a model derivative and is not an official Google model. Google has not endorsed this model.
Training
- Method: full-parameter DPO (no PEFT/LoRA)
- Training data:
argilla/dpo-mix-7k(MIT) - Epoch: 2
- Learning rate:
5e-7 - DPO beta:
0.1 - Loss: sigmoid
- Maximum sequence length: 4096
- Seed: 42
- Frameworks: Transformers and TRL
The training prompt used John as the user role tag and the system prompt
You are a helpful assistant for John.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "iamhyunmin/gemma2-9b-deita-sft-epoch3-dpo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
messages = [
{"role": "user", "content": "Explain direct preference optimization."}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
License and use restrictions
This model is a Gemma Model Derivative and is distributed under and subject to
the Gemma Terms of Use, including the
Gemma Prohibited Use Policy.
By downloading, using, modifying, or distributing this model, you agree to
those terms and restrictions. See NOTICE.
Limitations
This model may produce inaccurate, biased, unsafe, or otherwise undesirable content. It has not been evaluated for every downstream application. Users are responsible for evaluating the model and complying with applicable laws and the Gemma Terms of Use.