dpo-qwen2.5-0.5b-halueval

This model is part of the project Domain-Specific Hallucination Detection in Large Language Models (paper). The code and full pipeline are available on GitHub.

A Direct Preference Optimization (DPO) fine-tune of Qwen2.5-0.5B-Instruct trained to reduce hallucination across grounded QA, dialogue, and summarization tasks. Fine-tuned on preference pairs derived from all three subtasks of the HaluEval benchmark.

This model is the mitigation component of a complete hallucination detection + mitigation pipeline built for CS 593 NLP (Purdue University Fort Wayne, Spring 2026). Hallucination rates are measured by passing model generations through a separately fine-tuned DeBERTa detector: varunchundru/hallucination-detector-deberta.

Model Details

Training Details

Preference Pair Construction

Each HaluEval example was converted into a DPO triplet using task-specific formatting:

System prompt (QA / Dialogue):

You are a helpful assistant. Answer based only on the provided knowledge.
Do not invent facts. Be concise.

System prompt (Summarization):

Summarize the following document accurately.
Do not add information not present in the document.

Hyperparameters

Evaluation

Hallucination rates are measured on the held-out 15% test split (4,500 examples, 1,500 per task) by passing each model's generation through varunchundru/hallucination-detector-deberta at a 0.5 probability threshold.

Overall Results

Per-Task Breakdown

Notable findings:

• QA sees the largest absolute improvement — the structured knowledge + question format aligns well with DPO training signal.
• Summarization hallucination is effectively eliminated on the test set (0.0% rate), likely because the DPO training directly contrasts faithful vs. hallucinated summaries on similar documents.
• Dialogue shows minimal improvement (−5.6%). The model still hallucinates in 94.2% of dialogue turns, suggesting that multi-turn conversation is a harder distribution to shift with preference learning at this scale.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "varunchundru/dpo-qwen2.5-0.5b-halueval"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()


def answer_grounded(question: str, knowledge: str, max_new_tokens: int = 150) -> str:
    messages = [
        {
            "role": "system",
            "content": (
                "You are a helpful assistant. Answer based only on the provided knowledge. "
                "Do not invent facts. Be concise."
            ),
        },
        {
            "role": "user",
            "content": f"Knowledge: {knowledge}

Question: {question}",
        },
    ]
    text = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )
    response = tokenizer.decode(
        output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
    )
    return response.strip()


# Example
knowledge = "The Eiffel Tower is located in Paris, France, and was completed in 1889."
question = "Where is the Eiffel Tower located and when was it finished?"
print(answer_grounded(question, knowledge))

Intended Use & Limitations

Intended use:

• Grounded question answering and document summarization where faithfulness to a context is required
• Research on hallucination mitigation via preference learning
• Mitigation component paired with varunchundru/hallucination-detector-deberta

Limitations:

• Dialogue performance is weak — DPO training did not meaningfully reduce hallucination for multi-turn dialogue (94.2% post-DPO rate). The dialogue task may require more training, a larger model, or task-specific preference data.
• Trained for only 1 epoch on 0.5B parameters — further training or a larger base model would likely improve results.
• Hallucination rates are measured by a proxy DeBERTa classifier, not human annotation.
• max_length=512 during DPO training may truncate long documents in the summarization task.
• Should not be used in high-stakes domains without further validation.

Project Context

This model is the mitigation component in a four-part hallucination pipeline:

1. TF-IDF + Logistic Regression — Lightweight lexical baseline
2. Zero-Shot DeBERTa-MNLI — NLI-based detection without task-specific training (Acc=0.60, F1=0.43, AUROC=0.65)
3. Fine-Tuned DeBERTa-v3-base — Task-specific hallucination detector (Acc=0.91, F1=0.91, AUROC=0.98)
4. DPO Fine-Tuned Qwen2.5-0.5B (this model) — Reduces hallucination at generation time

Authors: Varun Chundru & Debasmita Biswas Course: CS 593 Natural Language Processing, Purdue University Fort Wayne, Spring 2026

Citation

@inproceedings{li2023halueval,
  title={HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models},
  author={Li, Junyi and Cheng, Xiaoxue and Zhao, Wayne Xin and Nie, Jian-Yun and Wen, Ji-Rong},
  booktitle={Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year={2023}
}