Nepali LLaMA 590M — 800M Token Exposure
This is a 590M-parameter Nepali-oriented causal language model continued from:
Anurag9817/nepali-llama-590m-580m-tokens
Training
Initial checkpoint exposure: ~580M tokens
Continuation data: 220M tokenizer tokens from FineWeb-Edu
Total approximate exposure: ~800M tokens
Architecture
- Parameters: 590,077,440
- Vocabulary size: 40,000
- Hidden size: 1536
- Intermediate size: 4096
- Layers: 18
- Attention heads: 12
- KV heads: 6
- Sequence length: 1024
Continuation setup
- 2 × Tesla T4
- Distributed Data Parallel (DDP)
- FP16
- 8-bit AdamW
- Gradient checkpointing: OFF
- torch.compile: ON
- Learning rate: 3e-4
- Warmup steps: 150
- Weight decay: 0.1
- Cosine scheduler
Tokenizer
The original raw SentencePiece tokenizer is included:
nepali_llama_40k.model
The tokenizer uses:
- vocab size: 40,000
- PAD = 0
- UNK = 1
- BOS = 2
- EOS = 3
Note
This checkpoint was produced by continued causal-language-model pretraining. It is not instruction-tuned / SFT-tuned.
The raw SentencePiece tokenizer should be used directly rather than replacing it with AutoTokenizer.