Qwen2.5-0.5B-Instruct + DPO (LoRA)
Fine-tuned from Qwen/Qwen2.5-0.5B-Instruct with DPO (LoRA r=16, beta=0.1, 1 epoch, 2,000 pairs from UltraFeedback) on a Kaggle T4 GPU. The LoRA adapter is merged into the weights.
Results (vs. the base model)
| Metric | Base | This model |
|---|---|---|
| TruthfulQA mc2 (200 questions) | 0.417 ± 0.029 | 0.414 ± 0.028 |
| Safety refusals (keyword, of 10) | 8 | 8 |
| Average answer length (words) | 109 | 110 |
DPO caused no measurable change. This is a learning project.
Limitations
- Small model: it makes up facts and is outdated on recent events.
- Weak in Bengali and other low-resource languages.
- Evaluation is small (200 questions, 30 prompts, keyword refusal check).
Usage
Load the tokenizer from the base model: ```python from transformers import pipeline pipe = pipeline("text-generation", model="Jubel/qwen2.5-0.5b-dpo", tokenizer="Qwen/Qwen2.5-0.5B-Instruct") ```
Code: https://github.com/JubelHossaain/dpo-aligned-assistant Base model credit: Qwen team, Alibaba Cloud.