mistral-7b-orpo-airoboros-pref-10k
This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the alvarobartt/airoboros2.2-pref-10k dataset. It achieves the following results on the evaluation set:
- Loss: 0.9271
- Rewards/chosen: -0.0459
- Rewards/rejected: -0.0501
- Rewards/accuracies: 0.5938
- Rewards/margins: 0.0041
- Logps/rejected: -1.0013
- Logps/chosen: -0.9186
- Logits/rejected: -2.7246
- Logits/chosen: -2.7340
- Nll Loss: 0.8613
- Log Odds Ratio: -0.7717
- Log Odds Chosen: 0.1600
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- total_train_batch_size: 32
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: inverse_sqrt
- lr_scheduler_warmup_ratio: 0.1
- lr_scheduler_warmup_steps: 100
- num_epochs: 3
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | Nll Loss | Log Odds Ratio | Log Odds Chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.7662 | 0.34 | 100 | 0.7563 | -0.0402 | -0.0436 | 0.6094 | 0.0033 | -0.8714 | -0.8045 | -2.7457 | -2.7631 | 0.7061 | -0.6883 | 0.1361 |
| 0.7165 | 0.67 | 200 | 0.7470 | -0.0379 | -0.0408 | 0.6016 | 0.0029 | -0.8160 | -0.7582 | -2.6133 | -2.6317 | 0.6912 | -0.6962 | 0.1223 |
| 0.6561 | 1.01 | 300 | 0.7483 | -0.0369 | -0.0388 | 0.5703 | 0.0019 | -0.7767 | -0.7384 | -2.5863 | -2.6061 | 0.6888 | -0.7299 | 0.0912 |
| 0.3724 | 1.35 | 400 | 0.7860 | -0.0386 | -0.0412 | 0.5859 | 0.0026 | -0.8244 | -0.7719 | -2.6543 | -2.6721 | 0.7220 | -0.7591 | 0.0882 |
| 0.3671 | 1.68 | 500 | 0.7863 | -0.0388 | -0.0426 | 0.5547 | 0.0038 | -0.8524 | -0.7761 | -2.7365 | -2.7521 | 0.7249 | -0.7034 | 0.1717 |
| 0.2292 | 2.02 | 600 | 0.8849 | -0.0434 | -0.0482 | 0.5781 | 0.0048 | -0.9642 | -0.8677 | -2.7897 | -2.8003 | 0.8235 | -0.7038 | 0.2164 |
| 0.1537 | 2.36 | 700 | 0.9065 | -0.0445 | -0.0497 | 0.5938 | 0.0051 | -0.9934 | -0.8905 | -2.6826 | -2.6902 | 0.8397 | -0.7166 | 0.2062 |
| 0.1664 | 2.69 | 800 | 0.8909 | -0.0445 | -0.0495 | 0.6172 | 0.0051 | -0.9909 | -0.8891 | -2.7237 | -2.7353 | 0.8254 | -0.7314 | 0.2106 |
Framework versions
- Transformers 4.39.0.dev0
- Pytorch 2.1.1+cu121
- Datasets 2.14.6
- Tokenizers 0.15.2