Nepali LLaMA 590M — English 180M
A 590M-parameter decoder-only LLaMA-family causal language model continued on 180M English tokens from FineWeb-Edu.
Model
- Parameters: 590,077,440
- Vocabulary size: 40,000
- Hidden size: 1536
- Intermediate size: 4096
- Layers: 18
- Attention heads: 12
- KV heads: 6
- Context length used during training: 1024
Training
- First stage: 100M FineWeb-Edu tokens
- Continuation: additional 80M FineWeb-Edu tokens
- Total English exposure: 180M tokens
- Sequence length: 1024
- Hardware: 2 × NVIDIA Tesla T4
- Precision: FP16
- Optimizer: AdamW 8-bit
- Gradient accumulation: 8
Tokenizer
This repository includes the project's raw SentencePiece tokenizer:
tokenizer/nepali_llama_40k.model
Tokenizer details:
- Vocab size: 40,000
- PAD = 0
- UNK = 1
- BOS = 2
- EOS = 3
The tokenizer is intentionally provided as the original SentencePiece model and should not be replaced with an unrelated AutoTokenizer.
Notes
This is a pretrained/continued-pretraining language model, not an instruction-tuned chat model.