final-ft__bert-base-spanish-wwm-uncased__70k-ultrasounds
This model is a fine-tuned version of dccuchile/bert-base-spanish-wwm-uncased on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.5422
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 256
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| No log | 0.9990 | 254 | 0.9709 |
| No log | 1.9980 | 508 | 0.8043 |
| No log | 2.9971 | 762 | 0.7331 |
| 1.013 | 4.0 | 1017 | 0.6860 |
| 1.013 | 4.9990 | 1271 | 0.6597 |
| 1.013 | 5.9980 | 1525 | 0.6345 |
| 1.013 | 6.9971 | 1779 | 0.6280 |
| 0.6672 | 8.0 | 2034 | 0.5954 |
| 0.6672 | 8.9990 | 2288 | 0.5987 |
| 0.6672 | 9.9980 | 2542 | 0.5802 |
| 0.6672 | 10.9971 | 2796 | 0.5792 |
| 0.6055 | 12.0 | 3051 | 0.5632 |
| 0.6055 | 12.9990 | 3305 | 0.5713 |
| 0.6055 | 13.9980 | 3559 | 0.5545 |
| 0.6055 | 14.9971 | 3813 | 0.5434 |
| 0.5735 | 16.0 | 4068 | 0.5475 |
| 0.5735 | 16.9990 | 4322 | 0.5471 |
| 0.5735 | 17.9980 | 4576 | 0.5377 |
| 0.5735 | 18.9971 | 4830 | 0.5403 |
| 0.5574 | 19.9803 | 5080 | 0.5422 |
Framework versions
- Transformers 4.40.1
- Pytorch 2.2.1+cu121
- Datasets 2.19.0
- Tokenizers 0.19.1