🌐 Company Website 🔗 Mozaic AI Solutions
✨ Overview
We were curious to see what happens if one uses:
$$
\text{{high-quality DPO dataset}} + \text{{merge of DPO optimized and non-DPO optimized model}}
$$
The underlying model used was:
/Weyaxi/OpenHermes-2.5-neural-chat-v3-3-Slerp
Dataset
Dataset: /argilla/distilabel-intel-orca-dpo-pairs
The dataset was roughly ~3000 samples but they were high quality (according to the chosen_score).
The following filters were applied to the original dataset:
dataset = dataset.filter(
lambda r:
r["status"] != "tie" and
r["chosen_score"] >= 8 and
not r["in_gsm8k_train"]
)
Chat Template
I decided to go with the ChatML which is used for OpenHermes2.5 By the way I integreated the chat template into the models tokenizer.
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{asistant}<|im_end|>
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 71.71 |
| AI2 Reasoning Challenge (25-Shot) | 68.94 |
| HellaSwag (10-Shot) | 86.45 |
| MMLU (5-Shot) | 63.97 |
| TruthfulQA (0-shot) | 64.01 |
| Winogrande (5-shot) | 79.95 |
| GSM8k (5-shot) | 66.94 |