Mazgu_Small-T_130M_Instruct
This model was trained from scratch on the None dataset. It achieves the following results on the evaluation set:
- Loss: 3.1128
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 0.03
- num_epochs: 22
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 3.1018 | 1.0 | 214 | 3.1789 |
| 3.0351 | 2.0 | 428 | 3.1612 |
| 3.0528 | 3.0 | 642 | 3.1474 |
| 2.9466 | 4.0 | 856 | 3.1380 |
| 2.9969 | 5.0 | 1070 | 3.1299 |
| 2.9784 | 6.0 | 1284 | 3.1246 |
| 2.9222 | 7.0 | 1498 | 3.1206 |
| 2.9501 | 8.0 | 1712 | 3.1179 |
| 3.0095 | 9.0 | 1926 | 3.1164 |
| 2.9389 | 10.0 | 2140 | 3.1149 |
| 2.9016 | 11.0 | 2354 | 3.1139 |
| 2.8938 | 12.0 | 2568 | 3.1136 |
| 2.8549 | 13.0 | 2782 | 3.1131 |
| 2.8796 | 14.0 | 2996 | 3.1130 |
| 2.9367 | 15.0 | 3210 | 3.1129 |
| 2.8664 | 16.0 | 3424 | 3.1128 |
| 2.8942 | 17.0 | 3638 | 3.1129 |
| 2.8932 | 18.0 | 3852 | 3.1127 |
| 2.8682 | 19.0 | 4066 | 3.1127 |
| 2.9357 | 20.0 | 4280 | 3.1128 |
| 2.9435 | 21.0 | 4494 | 3.1128 |
Framework versions
- Transformers 5.0.0
- Pytorch 2.10.0+cu128
- Datasets 5.0.0
- Tokenizers 0.22.2