distilroberta-rb156k-opt15-ep40
This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 1.2088
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 7e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 40
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.3431 | 1.0 | 8994 | 1.3416 |
| 1.3385 | 2.0 | 17988 | 1.3477 |
| 1.3321 | 3.0 | 26982 | 1.3444 |
| 1.3277 | 4.0 | 35976 | 1.3370 |
| 1.315 | 5.0 | 44970 | 1.3325 |
| 1.3081 | 6.0 | 53964 | 1.3314 |
| 1.3056 | 7.0 | 62958 | 1.3271 |
| 1.2999 | 8.0 | 71952 | 1.3219 |
| 1.2973 | 9.0 | 80946 | 1.3195 |
| 1.2865 | 10.0 | 89940 | 1.3135 |
| 1.2749 | 11.0 | 98934 | 1.3111 |
| 1.2685 | 12.0 | 107928 | 1.3077 |
| 1.2652 | 13.0 | 116922 | 1.2986 |
| 1.2604 | 14.0 | 125916 | 1.2979 |
| 1.255 | 15.0 | 134910 | 1.2942 |
| 1.2508 | 16.0 | 143904 | 1.2870 |
| 1.2358 | 17.0 | 152898 | 1.2858 |
| 1.2361 | 18.0 | 161892 | 1.2825 |
| 1.2302 | 19.0 | 170886 | 1.2741 |
| 1.2242 | 20.0 | 179880 | 1.2743 |
| 1.2177 | 21.0 | 188874 | 1.2665 |
| 1.2055 | 22.0 | 197868 | 1.2642 |
| 1.2096 | 23.0 | 206862 | 1.2583 |
| 1.2065 | 24.0 | 215856 | 1.2567 |
| 1.1931 | 25.0 | 224850 | 1.2537 |
| 1.1908 | 26.0 | 233844 | 1.2510 |
| 1.1875 | 27.0 | 242838 | 1.2480 |
| 1.1803 | 28.0 | 251832 | 1.2424 |
| 1.1711 | 29.0 | 260826 | 1.2394 |
| 1.173 | 30.0 | 269820 | 1.2380 |
| 1.1665 | 31.0 | 278814 | 1.2291 |
| 1.1586 | 32.0 | 287808 | 1.2291 |
| 1.1607 | 33.0 | 296802 | 1.2260 |
| 1.1524 | 34.0 | 305796 | 1.2228 |
| 1.1463 | 35.0 | 314790 | 1.2171 |
| 1.1453 | 36.0 | 323784 | 1.2135 |
| 1.1387 | 37.0 | 332778 | 1.2135 |
| 1.136 | 38.0 | 341772 | 1.2117 |
| 1.1347 | 39.0 | 350766 | 1.2070 |
| 1.1347 | 40.0 | 359760 | 1.2078 |
Framework versions
- Transformers 4.35.0
- Pytorch 2.1.0+cu118
- Datasets 2.14.6
- Tokenizers 0.14.1