Model Information
Model Details
Model Description
Llama3-ViettelSolutions-8B is a variant of the Meta Llama-3-8B model, continued pre-trained on the Vietnamese curated dataset and supervised fine-tuned on 5 million samples of Vietnamese instruct data.
- Developed by: Viettel Solutions
- Funded by: NVIDIA
- Model type: Autoregressive transformer model
- Language(s) (NLP): Vietnamese, English
- License: Llama 3 Community License
- Finetuned from model: meta-llama/Meta-Llama-3-8B
Uses
Example snippet for usage with Transformers:
import transformers
import torch
model_id = "VTSNLP/Llama3-ViettelSolutions-8B"
pipeline = transformers.pipeline(
"text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto"
)
pipeline("Xin chào!")
Training Details
Training Data
-
Dataset for continue pretrain: Vietnamese curated dataset
-
Dataset for supervised fine-tuning: Instruct general dataset
Training Procedure
Preprocessing
[More Information Needed]
Training Hyperparameters
- Training regime: bf16 mixed precision
- Data sequence length: 8192
- Tensor model parallel size: 4
- Pipelinemodel parallel size: 1
- Context parallel size: 1
- Micro batch size: 1
- Global batch size: 512
Evaluation
Testing Data, Factors & Metrics
Testing Data
[More Information Needed]
Factors
[More Information Needed]
Metrics
[More Information Needed]
Results
[More Information Needed]
Summary
[More Information Needed]
Technical Specifications
-
Compute Infrastructure: NVIDIA DGX
-
Hardware: 4 x A100 80GB
-
Software: NeMo Framework
Citation
BibTeX:
[More Information Needed]
APA:
[More Information Needed]
More Information
[More Information Needed]
Model Card Authors
[More Information Needed]
Model Card Contact
[More Information Needed]