Nepali LLaMA 590M — 1.75B Tokens
A 590M-parameter LLaMA-style causal language model continued-pretrained on FineWeb-Edu.
Model
- Parameters: 590,077,440
- Vocabulary: 40,000
- Hidden size: 1536
- Intermediate size: 4096
- Layers: 18
- Attention heads: 12
- Key/Value heads: 6
- Maximum position embeddings: 2048
Training
Base checkpoint:
Anurag9817/nepali-llama-590m-1.50b-tokens
Dataset:
HuggingFaceFW/fineweb_edu_100BT-shuffled
- First 500M tokenizer tokens skipped
- Next 250,085,376 tokenizer tokens trained
- Sequence length: 1024
- 2x NVIDIA T4
- Batch/GPU: 2
- Gradient accumulation: 64
- Global tokens/update: 262,144
- Optimizer steps: 954
- Learning rate: 1e-4
- Warmup: 0
- Weight decay: 0.1
- Scheduler: cosine
- FP16: enabled
- 8-bit AdamW: enabled
- Gradient checkpointing: disabled
- torch.compile: enabled
- Compile mode: default
- DataLoader workers: 2/GPU
- No intermediate checkpoints
Result
- Runtime: 10.17 hours
- Final training loss: 2.80852
- Additional tokens: 250,085,376
- Approximate cumulative exposure: 1.75B tokens
Tokenizer
Original SentencePiece tokenizer:
tokenizer/nepali_llama_40k.model
Special token IDs:
- PAD = 0
- UNK = 1
- BOS = 2
- EOS = 3
The original tokenizer must not be replaced or resized.
Tokenizer Usage
import sentencepiece as spm
sp = spm.SentencePieceProcessor(
model_file="tokenizer/nepali_llama_40k.model"
)
text = "नेपाल एक सुन्दर देश हो।"
ids = sp.encode(text, out_type=int)
decoded = sp.decode(ids)
print(ids)
print(decoded)
Important
This checkpoint is continued-pretraining only.
It has not been instruction-tuned or SFT-trained.