Nepali LLaMA 590M — ~352M Token Model
Custom 590,077,440-parameter LLaMA-style causal language model.
Model
- Parameters: 590,077,440
- Vocabulary: 40,000
- Context length: 2,048
- Hidden size: 1,536
- Layers: 18
- Attention heads: 12
- KV heads: 6
Training exposure
Approximately 170M Nepali tokens followed by 181,011,456 mixed-language tokens.
Approximate total exposure:
~351M tokens.
Mixed continuation dataset
- 80M Devanagari Nepali
- 80M English
- 22,290,432 Roman Nepali
Tokenizer
Custom 40K SentencePiece tokenizer:
tokenizer/nepali_llama_40k.model
Special token IDs:
- PAD = 0
- UNK = 1
- BOS = 2
- EOS = 3
For exact compatibility, use the supplied tokenizer.
Continue training
The model can be loaded with:
from transformers import LlamaForCausalLM
model = LlamaForCausalLM.from_pretrained( "Anurag9817/nepali-llama-590m-352m-mixed" )
Training configuration is stored in:
training_config.json
Package contents
- model weights
- config.json
- generation_config.json
- tokenizer/nepali_llama_40k.model
- tokenizer/tokenizer_config.json
- tokenizer/special_tokens_map.json
- training_config.json
- README.md