Millesime-202608-4B-phase2
This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 on the comparia_dpo dataset. It achieves the following results on the evaluation set:
- Loss: 0.8321
- Rewards/chosen: 1.1462
- Rewards/rejected: -3.0919
- Rewards/accuracies: 0.7660
- Rewards/margins: 4.2381
- Logps/chosen: -1016.5903
- Logps/rejected: -1058.3787
- Logits/chosen: -3.0804
- Logits/rejected: -3.1843
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- total_eval_batch_size: 4
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 0.1
- num_epochs: 1.0
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/chosen | Logps/rejected | Logits/chosen | Logits/rejected |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.7084 | 0.1055 | 50 | 0.5794 | -1.2842 | -3.4253 | 0.7372 | 2.1411 | -1040.8944 | -1061.7122 | -2.9093 | -3.0172 |
| 1.3953 | 0.2110 | 100 | 0.9903 | -0.9245 | -5.2865 | 0.7340 | 4.3620 | -1037.2972 | -1080.3245 | -3.0515 | -3.1622 |
| 1.1409 | 0.3165 | 150 | 1.0129 | 1.0182 | -2.7772 | 0.7468 | 3.7955 | -1017.8700 | -1055.2316 | -3.0271 | -3.1316 |
| 1.2015 | 0.4221 | 200 | 1.0720 | 0.2541 | -4.8643 | 0.7468 | 5.1184 | -1025.5115 | -1076.1021 | -2.9604 | -3.0636 |
| 1.4030 | 0.5276 | 250 | 1.0206 | 2.6027 | -1.4402 | 0.7276 | 4.0430 | -1002.0249 | -1041.8613 | -2.9602 | -3.0647 |
| 2.0725 | 0.6331 | 300 | 0.9071 | -0.4076 | -5.2369 | 0.7788 | 4.8293 | -1032.1279 | -1079.8281 | -3.0332 | -3.1352 |
| 0.5974 | 0.7386 | 350 | 0.8837 | 0.7087 | -3.8638 | 0.7821 | 4.5724 | -1020.9659 | -1066.0969 | -3.0906 | -3.1961 |
| 0.7703 | 0.8441 | 400 | 0.8303 | 0.3155 | -4.1147 | 0.7821 | 4.4302 | -1024.8973 | -1068.6063 | -3.0833 | -3.1875 |
| 0.8016 | 0.9496 | 450 | 0.8429 | 1.1231 | -3.0798 | 0.7756 | 4.2030 | -1016.8212 | -1058.2578 | -3.0821 | -3.1860 |
| 0.7013 | 1.0 | 474 | 0.8321 | 1.1462 | -3.0919 | 0.7660 | 4.2381 | -1016.5903 | -1058.3787 | -3.0804 | -3.1843 |
Framework versions
- Transformers 5.8.0
- Pytorch 2.13.0+cu130
- Datasets 4.0.0
- Tokenizers 0.22.2