llama-256-12L-qa
This model is a fine-tuned version of stage-babylm/llama-256-12L on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 1.7426
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 0.1
- num_epochs: 3
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 2.6367 | 0.1507 | 171 | 2.5935 |
| 2.0299 | 0.3013 | 342 | 1.9835 |
| 1.8834 | 0.4520 | 513 | 1.8715 |
| 1.8843 | 0.6026 | 684 | 1.8319 |
| 1.7929 | 0.7533 | 855 | 1.8088 |
| 1.8340 | 0.9040 | 1026 | 1.7920 |
| 1.6984 | 1.0546 | 1197 | 1.7805 |
| 1.7795 | 1.2053 | 1368 | 1.7718 |
| 1.7848 | 1.3559 | 1539 | 1.7657 |
| 1.7373 | 1.5066 | 1710 | 1.7593 |
| 1.7183 | 1.6573 | 1881 | 1.7548 |
| 1.7889 | 1.8079 | 2052 | 1.7509 |
| 1.6819 | 1.9586 | 2223 | 1.7478 |
| 1.7535 | 2.1093 | 2394 | 1.7460 |
| 1.6669 | 2.2599 | 2565 | 1.7448 |
| 1.7614 | 2.4106 | 2736 | 1.7437 |
| 1.6935 | 2.5612 | 2907 | 1.7431 |
| 1.7186 | 2.7119 | 3078 | 1.7428 |
| 1.6719 | 2.8626 | 3249 | 1.7426 |
| 1.6835 | 3.0 | 3405 | 1.7426 |
Framework versions
- Transformers 5.14.1
- Pytorch 2.13.0+cu130
- Datasets 5.0.0
- Tokenizers 0.22.2