Nepali LLaMA 590M — 200M Nepali CPT
590M parameter LLaMA-style causal language model.
Model
- Parameters: 590,077,440
- Vocabulary size: 40,000
- Context length: 1,024
- Continued pretraining tokens: 199,983,104
- Training steps: 6,103
- FP16: yes
- 8-bit AdamW: yes
- SDPA: yes
- Gradient checkpointing: no
Tokenizer
The original 40,000-token SentencePiece tokenizer is provided
as tokenizer.model.
- PAD: 0 (
<pad>) - UNK: 1 (
<unk>) - BOS: 2 (
<s>) - EOS: 3 (
</s>)
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "Anurag9817/nepali-llama-590m-200m-nepali-cpt"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "नेपाल एक सुन्दर र विविधताले भरिएको देश हो।"
inputs = tokenizer(prompt, return_tensors='pt')
outputs = model.generate(
**inputs,
max_new_tokens=100,
do_sample=True,
temperature=0.8,
top_p=0.9,
repetition_penalty=1.05,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))