qa-finetune
This model is a fine-tuned version of stage-babylm/llama-256-12L on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 1.9113
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 0.1
- num_epochs: 3
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 2.6867 | 0.1901 | 100 | 2.7004 |
| 2.1401 | 0.3802 | 200 | 2.1452 |
| 2.0333 | 0.5703 | 300 | 2.0345 |
| 1.9394 | 0.7605 | 400 | 1.9897 |
| 1.9388 | 0.9506 | 500 | 1.9645 |
| 1.9553 | 1.1407 | 600 | 1.9493 |
| 1.9092 | 1.3308 | 700 | 1.9371 |
| 1.8855 | 1.5209 | 800 | 1.9292 |
| 1.8996 | 1.7110 | 900 | 1.9226 |
| 1.9202 | 1.9011 | 1000 | 1.9182 |
| 1.8683 | 2.0913 | 1100 | 1.9149 |
| 1.8255 | 2.2814 | 1200 | 1.9132 |
| 1.8680 | 2.4715 | 1300 | 1.9121 |
| 1.8863 | 2.6616 | 1400 | 1.9115 |
| 1.8532 | 2.8517 | 1500 | 1.9113 |
| 1.8478 | 3.0 | 1578 | 1.9113 |
Framework versions
- Transformers 5.14.1
- Pytorch 2.13.0+cu130
- Datasets 5.0.0
- Tokenizers 0.22.2