MiniCPM5-2B Nepali — Agriculture
Full-weight supervised fine-tune of openbmb/MiniCPM5-2B on Nepali instruction data.
Phase directory on the Hub: /.
Generated 2026-10-03 10:54 UTC.
Base model
| Base | openbmb/MiniCPM5-2B |
| Architecture | LlamaForCausalLM, 42 layers, hidden 2048, GQA (16 heads / 2 KV) |
| Training type | full-weight SFT (not LoRA) |
| Precision | bfloat16 |
| Context length | 4096 tokens (native 131072; no scaling applied) |
The base model is a thinking model: its chat template injects an empty
<think></think> block before every assistant turn. Training reproduces that
token sequence exactly, so the tuned model stays compatible with the stock
template and with vLLM/SGLang/llama.cpp chat formatting.
Data
| Split | Records |
|---|---|
| train | 20,987 |
| validation | 2,700 |
| test | 2,609 |
Sources, both Nepali (Devanagari):
himalaya-ai/unified-sft-dataset— aggregated Nepali SFT; per-record upstream licences range over CC-BY-4.0 and other permissive terms (license_tier: permissive), carried in the source metadata.aarajbhattarai/nepali-agri-gov-instruct— agriculture/government/law instructions with grounding, correctness and naturalness judge scores.
Cleaning
Records whose assistant turn contained an empty fact slot were dropped: templated school-directory answers such as
Himali Secondary School को सम्पर्क नम्बर हो। स्थापना मा भएको हो।
where the phone number and year are literally absent. Training on these teaches the model to assert facts it was never given, so they are removed rather than down-weighted — the sentence pattern is the label and there is no correct target to learn. This removed 13.75% of the raw training split (5,551 records).
Splits were verified to be disjoint: zero duplicate conversations within or across train/validation/test.
Hyperparameters
| Epochs | 3 |
| Learning rate | 8e-6 (cosine, 3% warmup) |
| Effective batch | 32 sequences (4 x 8 accumulation) |
| Max sequence length | 4096 |
| Weight decay | 0.01 |
| Grad clipping | 1.0 |
| Loss | assistant tokens only |
| Optimizer | AdamW |
max_length was chosen from the measured token distribution rather than a
round number: at 4096 about 92% of all training tokens are retained
(p95 = 3010 tokens for the broad split).
Batch size was set from a worst-case VRAM probe — a full batch of maximum-length sequences — not from an average batch. On this H100 NVL (95 GB) batch 4 x 4096 OOMs while batch 2 x 4096 peaks at 57.4 GB, so throughput is bought with gradient accumulation instead.
Metrics
| metric | value |
|---|---|
epoch | 3.0000 |
eval_loss | 0.8771 |
eval_mean_token_accuracy | 0.7573 |
eval_num_tokens | 152072076.0000 |
eval_runtime | 70.2091 |
eval_samples_per_second | 17.2060 |
eval_steps_per_second | 4.3010 |
Checkpointing and retention
Checkpoints are saved every 250 steps and at each epoch end, and uploaded to the Hub from a background thread (training never blocks on, or fails from, an upload).
Each checkpoint contains model weights, trainer state, RNG state and the exact config. Optimizer and scheduler state are kept local only — they are roughly twice the model size, are useful solely for resuming (which happens from local disk), and would dominate the repository.
Retention: the best checkpoint by validation loss (/best/) plus the
last 2 step checkpoints. The brief's "every 5th checkpoint" research-milestone
rule applies to small LoRA adapters; this is full-weight training where every
checkpoint is ~5 GB, so that tier is dropped as the brief specifies.
Training auto-resumes from the newest local checkpoint if the process restarts.
Known limitations
- The corpus is heavily weighted toward templated factual QA (school and institution directories). Expect strong recall of that register and weaker open-ended generation.
- Blank-slot filtering removes the worst hallucination teacher but cannot verify that the remaining facts are correct; the directory data was never independently validated.
- No safety / refusal tuning was performed.
- No OCR or vision capability: this is a text-only model.
- Evaluation is validation/test loss on in-distribution data. There is no external Nepali benchmark score here, so these numbers do not establish general Nepali ability.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "aarajbhattarai/minicpm5-2b-nepali-agriculture" # subfolder="/best"
tok = AutoTokenizer.from_pretrained(model_id, subfolder="/best")
model = AutoModelForCausalLM.from_pretrained(
model_id, subfolder="/best", dtype=torch.bfloat16, device_map="auto").eval()
msgs = [{"role": "user", "content": "नेपालको राजधानी कहाँ हो?"}]
inputs = tok.apply_chat_template(
msgs, add_generation_prompt=True, enable_thinking=False,
return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))