T-Llama-3-8B (October 2024)
Korean Continual Pre-trained Model based on Meta-Llama-3-8B
Model Description
- Developed by: TmaxAI
- Shared by: MinJae Kim (mjk00098@gmail.com)
- License: cc-by-nc-nd-4.0
- Data: 64GB of Korean Corpus (Trained About 18.8B Tokens)
- Sequence Length: 8k
- Resource: H100 80GB x 8
Accelerating
To optimize performance and handle large-scale Korean datasets efficiently, we integrated cutting-edge optimization techniques:
- DeepSpeed: Advanced distributed training framework for memory and computational efficiency.
- Gradient Checkpointing: Reduces memory usage during backpropagation by saving only necessary states.
- Flash_Attention_2: Significantly accelerates attention mechanisms with reduced memory overhead.
- BF16 Mixed Precision: Ensures fast computations with high numerical stability.
Evaluation (In Progress)
- Translation (MMT)
- Fine-tuned with PEFT on a multilingual translation task by freezing PLMs, and evaluated using xCOMET scores.
- Our models significantly outperformed the original PLM and achieved performance comparable to leading publicly available Korean continuous pretraining models.