malayalam-moe-pretrain
This model is a fine-tuned version of siyah1/malayalam-moe-pretrain on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 1.0009
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 64
- total_train_batch_size: 256
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 0.02
- num_epochs: 5
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 95.2033 | 1.2500 | 500 | 1.4825 |
| 81.1107 | 2.5000 | 1000 | 1.3183 |
| 79.8998 | 3.0 | 1200 | 1.3110 |
| 78.8520 | 3.7500 | 1500 | 1.2993 |
| 72.8643 | 5.0 | 2000 | 1.2758 |
| 68.8933 | 6.0 | 2400 | 1.2886 |
| 70.5932 | 6.2500 | 2500 | 1.3040 |
| 65.6363 | 7.5000 | 3000 | 1.3147 |
| 60.5359 | 8.7500 | 3500 | 1.3476 |
| 60.8426 | 9.0 | 3600 | 1.3474 |
| 79.4108 | 1.3464 | 4000 | 1.2411 |
| 77.1449 | 1.5147 | 4500 | 1.1966 |
| 88.9392 | 0.8473 | 5000 | 1.3731 |
| 76.6402 | 0.9320 | 5500 | 1.1977 |
| 73.9258 | 0.4989 | 6000 | 1.1527 |
| 71.7222 | 0.5405 | 6500 | 1.1238 |
| 70.6645 | 0.5821 | 7000 | 1.1032 |
| 70.0252 | 0.6237 | 7500 | 1.0877 |
| 68.4742 | 0.6652 | 8000 | 1.0745 |
| 68.2024 | 0.7068 | 8500 | 1.0635 |
| 67.3756 | 0.7484 | 9000 | 1.0541 |
| 67.3787 | 0.7900 | 9500 | 1.0454 |
| 66.5043 | 0.8316 | 10000 | 1.0377 |
| 65.9830 | 0.8731 | 10500 | 1.0311 |
| 65.4952 | 0.9147 | 11000 | 1.0252 |
| 65.4397 | 0.9563 | 11500 | 1.0196 |
| 64.6877 | 0.9979 | 12000 | 1.0133 |
| 64.1041 | 1.0394 | 12500 | 1.0099 |
| 63.9178 | 1.0810 | 13000 | 1.0073 |
| 64.0613 | 1.1226 | 13500 | 1.0040 |
| 63.8685 | 1.1642 | 14000 | 1.0009 |
Framework versions
- Transformers 5.17.0
- Pytorch 2.10.0+cu128
- Datasets 5.0.1
- Tokenizers 0.23.2