Nepali LLaMA 590M — 1.30B Token Exposure

A 590M-parameter LLaMA-style causal language model trained from scratch and continued with additional English-language causal pretraining.

This checkpoint is a base pretrained model and has not been instruction tuned or SFT trained.

Model Architecture

• Parameters: 590,077,440
• Vocabulary size: 40,000
• Hidden size: 1,536
• Intermediate size: 4,096
• Layers: 18
• Attention heads: 12
• Key/value heads: 6
• Maximum configured position embeddings: 2,048

Latest Continued Pretraining

Base checkpoint:

Anurag9817/nepali-llama-590m-1.05b-tokens

Latest CPT stage:

• Dataset: HuggingFaceFW/fineweb_edu_100BT-shuffled
• Additional tokens: 250,019,840
• Sequence length: 1,024
• 2x Tesla T4
• Batch per GPU: 2
• Gradient accumulation: 8
• Global tokens/update: 32,768
• Optimizer: AdamW 8-bit
• Learning rate: 3e-4
• Warmup steps: 150
• Weight decay: 0.1
• Scheduler: cosine
• FP16: enabled
• Gradient checkpointing: disabled
• torch.compile: enabled
• Workers/GPU: 4
• Intermediate checkpoints: disabled
• Final model only: enabled

Latest training result:

• Training loss: 2.9864
• Runtime: ~10.85 hours
• Optimizer steps: 7,630

Approximate cumulative pretraining exposure after this stage:

~1.30B tokens

Tokenizer

The model uses the original 40,000-token SentencePiece tokenizer:

tokenizer/nepali_llama_40k.model

Special token IDs:

• PAD: 0
• UNK: 1
• BOS: 2
• EOS: 3

The tokenizer was kept unchanged during continued pretraining.

The repository also includes:

tokenizer_helper.py

This helper loads and validates the original tokenizer.

Loading the Model

Because the original tokenizer is a raw SentencePiece model, use SentencePiece directly:

import sentencepiece as spm
from transformers import AutoModelForCausalLM
import torch

repo_id = "Anurag9817/nepali-llama-590m-1.30b-tokens"

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.float16,
)

sp = spm.SentencePieceProcessor(
    model_file="tokenizer/nepali_llama_40k.model"
)

prompt = "Nepal is"

input_ids = sp.encode(
    prompt,
    out_type=int,
)

print(input_ids)

Model Status

This is a base language model, not a chat or instruction-tuned model.

There is no SFT at this checkpoint.

The model is still under continued-pretraining development, with future work focused on improving broad knowledge, reasoning, Nepali/Devanagari capability, and later instruction following.

Repository Contents

• model.safetensors
• config.json
• generation_config.json
• tokenizer/nepali_llama_40k.model
• tokenizer_helper.py
• tokenizer_info.json
• training_info.json
• training_metrics.json
• training/train_250m_ddp.py
• README.md