Model Card for Llama-3-OffsetBias-8B

Llama-3-OffsetBias-8B is a generative judge model that performs pairwise preference evaluation task. It is trained to be more robust on various evaluation biases commonly found in evaluation models. The model is introduced in paper OffsetBias: Leveraging Debiased Data for Tuning Evaluators.

Model Details

Model Description

Llama-3-OffsetBias-8B is built with Meta Llama-3-8B-Instruct. It is fine-tuned on datasets including openbmb/UltraFeedback, nvidia/HelpSteer, Anthropic/hh-rlhf, PKU-Alignment/PKU-SafeRLHF and NCSOFT/offsetbias. The training is done with instruction-tuning methodology, where the target task is pairwise preference evaluation, where Instruction, Output (a), Output (b) are given, and a better output to the instruction needs to be found. The input is formatted with a specific prompt template, and the model outputs "Output (a)" or "Output (b)" as a prediction for better response. The prompt is specified in the Uses section.

• Developed by: NC Research
• Language(s) (NLP): English
• License: META LLAMA 3 COMMUNITY LICENSE AGREEMENT
• Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct

Model Sources

• 💻 Repository: https://github.com/ncsoft/offsetbias
• 📜 Paper: OffsetBias: Leveraging Debiased Data for Tuning Evaluators
• 🤗 Dataset: https://huggingface.co/datasets/NCSOFT/offsetbias

Uses

Direct Use

Suppose you have an pairwise evaluation instance, a triplet of (instruction, output_a and output_b). Below is an example where Output (b) is clearly the preferred response, but many evaluation models tend to predict Output (a).

instruction = "explain like im 5"
output_a = "Scientists are studying special cells that could help treat a sickness called prostate cancer. They even tried these cells on mice and it worked!"
output_b = "Sure, I'd be happy to help explain something to you! What would you like me to explain?"

OffsetBias model is intended to use a specific prompt format. The filled out prompt is then formatted as user message in a conversation.

prompt_template = """You are a helpful assistant in evaluating the quality of the outputs for a given instruction. Your goal is to select the best output for the given instruction.
                
Select the Output (a) or Output (b) that is better for the given instruction. The two outputs are generated by two different AI chatbots respectively.
Do NOT provide any explanation for your choice.
Do NOT say both / neither are good.
You should answer using ONLY “Output (a)” or “Output (b)”. Do NOT output any other words.
Here are some rules of the evaluation:
(1) You should prioritize evaluating whether the output honestly/precisely/closely executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc.
(2) Outputs should NOT contain more/less than what the instruction asks for, as such outputs do NOT precisely execute the instruction.
(3) You should avoid any potential bias and your judgment should be as objective as possible. For example, the order in which the outputs were presented should NOT affect your judgment, as Output (a) and Output (b) are **equally likely** to be the better.

# Instruction:
{input}
# Output (a):
{output_1}
# Output (b):
{output_2}
# Which is better, Output (a) or Output (b)? Your response should be either “Output (a)” or “Output (b)”:"""

user_message = prompt_template.format(input=instruction, output_1=output_a, output_2=output_b)

conversation = [{"role": "user", "content": user_message}]

With conversation ready, you can input it into the model for inference. The model should output "Output (b)" to be correct.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "NCSOFT/Llama-3-OffsetBias-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

input_ids = tokenizer.apply_chat_template(
    conversation, 
    tokenize=True, 
    add_generation_prompt=True, 
    return_tensors="pt")

generation = model.generate(
    input_ids=input_ids,
    max_new_tokens=20,
    do_sample=False,
    pad_token_id=128009,
    temperature=0)

completion = tokenizer.decode(
    generation[0][len(input_ids[0]):], 
    skip_special_tokens=True, 
    clean_up_tokenization_spaces=True)

print(completion)
# The model should output "Output (b)"

Out-of-Scope Use

Model inputs that do not follow the specified prompt format are considered out-of-scope use. Custom input format can result in unintended text output and should be used at the user's own discretion.

Evaluation

LLMBar Result

EvalBiasBench Result

Citation

BibTeX:

@misc{park2024offsetbias,
      title={OffsetBias: Leveraging Debiased Data for Tuning Evaluators},
      author={Junsoo Park and Seungyeon Jwa and Meiying Ren and Daeyoung Kim and Sanghyuk Choi},
      year={2024},
      eprint={2407.06551},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}