fr_wiki_clm_42
This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 3.4716
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 40000
- training_steps: 100000
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| No log | 1.7190 | 2000 | 7.1152 |
| 7.1938 | 3.4379 | 4000 | 5.8793 |
| 7.1938 | 5.1569 | 6000 | 5.4285 |
| 5.4621 | 6.8758 | 8000 | 5.0639 |
| 5.4621 | 8.5948 | 10000 | 4.7716 |
| 4.8217 | 10.3137 | 12000 | 4.5288 |
| 4.8217 | 12.0327 | 14000 | 4.3214 |
| 4.3693 | 13.7516 | 16000 | 4.1482 |
| 4.3693 | 15.4706 | 18000 | 4.0090 |
| 4.0391 | 17.1895 | 20000 | 3.8879 |
| 4.0391 | 18.9085 | 22000 | 3.7858 |
| 3.7885 | 20.6274 | 24000 | 3.7020 |
| 3.7885 | 22.3464 | 26000 | 3.6289 |
| 3.5937 | 24.0653 | 28000 | 3.5639 |
| 3.5937 | 25.7843 | 30000 | 3.5148 |
| 3.4308 | 27.5032 | 32000 | 3.4743 |
| 3.4308 | 29.2222 | 34000 | 3.4371 |
| 3.2954 | 30.9411 | 36000 | 3.4091 |
| 3.2954 | 32.6601 | 38000 | 3.3876 |
| 3.1719 | 34.3790 | 40000 | 3.3724 |
| 3.1719 | 36.0980 | 42000 | 3.3519 |
| 3.0572 | 37.8169 | 44000 | 3.3406 |
| 3.0572 | 39.5359 | 46000 | 3.3352 |
| 2.9443 | 41.2548 | 48000 | 3.3352 |
| 2.9443 | 42.9738 | 50000 | 3.3257 |
| 2.8516 | 44.6927 | 52000 | 3.3270 |
| 2.8516 | 46.4117 | 54000 | 3.3365 |
| 2.7715 | 48.1306 | 56000 | 3.3443 |
| 2.7715 | 49.8496 | 58000 | 3.3433 |
| 2.699 | 51.5685 | 60000 | 3.3520 |
| 2.699 | 53.2875 | 62000 | 3.3642 |
| 2.6393 | 55.0064 | 64000 | 3.3667 |
| 2.6393 | 56.7254 | 66000 | 3.3729 |
| 2.578 | 58.4443 | 68000 | 3.3848 |
| 2.578 | 60.1633 | 70000 | 3.3957 |
| 2.5319 | 61.8823 | 72000 | 3.3952 |
| 2.5319 | 63.6012 | 74000 | 3.4042 |
| 2.4839 | 65.3202 | 76000 | 3.4178 |
| 2.4839 | 67.0391 | 78000 | 3.4186 |
| 2.4449 | 68.7581 | 80000 | 3.4301 |
| 2.4449 | 70.4770 | 82000 | 3.4361 |
| 2.4072 | 72.1960 | 84000 | 3.4462 |
| 2.4072 | 73.9149 | 86000 | 3.4504 |
| 2.3747 | 75.6339 | 88000 | 3.4536 |
| 2.3747 | 77.3528 | 90000 | 3.4626 |
| 2.3448 | 79.0718 | 92000 | 3.4635 |
| 2.3448 | 80.7907 | 94000 | 3.4663 |
| 2.3181 | 82.5097 | 96000 | 3.4693 |
| 2.3181 | 84.2286 | 98000 | 3.4723 |
| 2.2982 | 85.9476 | 100000 | 3.4716 |
Framework versions
- Transformers 4.45.2
- Pytorch 2.5.1+cu124
- Datasets 3.0.1
- Tokenizers 0.20.1