Nepali LLaMA 590M — 380M English
A 590M-parameter causal language model.
Architecture
- Parameters: 590,077,440
- Vocabulary: 40,000
- Hidden size: 1536
- Intermediate size: 4096
- Layers: 18
- Attention heads: 12
- KV heads: 6
Training
Base checkpoint:
Anurag9817/nepali-llama-590m-180m-english
Additional training:
- Previous exposure: 180M English tokens
- New exposure: 200M English tokens
- Total exposure: 380M English tokens
- Dataset: FineWeb-Edu
- Sequence length: 1024
- Batch/GPU: 2
- Gradient accumulation: 8
- Effective tokens/update: 32,768
- GPUs: 2 × Tesla T4
- Learning rate: 1e-5
- Warmup: 100
- Scheduler: cosine
- Optimizer: AdamW 8-bit
- FP16: enabled
- Gradient checkpointing: disabled
- Optimizer steps: 6104
- Final training loss: 4.196
- Training time: 8.82 hours
Tokenizer
Raw SentencePiece tokenizer.
File:
tokenizer/nepali_llama_40k.model
Special token IDs:
- PAD = 0
- UNK = 1
- BOS = 2
- EOS = 3
Do not replace the project tokenizer with AutoTokenizer.
Status
This is a base pretrained language model, not an instruction-tuned chatbot.