🧬 MiniCPM5-2B-SFT-Pashto-Instruct

Pashto Instruction-Tuned Model (Tokenizer Surgery + SFT) 🚀🧬🇦🇫

MiniCPM5-2B-SFT-Pashto-Instruct is a specialized Pashto instruction-tuned model developed as part of the iPashto.ai initiative.

Two-Stage Development Pipeline:

1. Stage 1: Tokenizer Surgery 🔧

   • Applied tokenizer surgery to openbmb/MiniCPM5-2B
   • Added 46 missing Pashto/Urdu characters as single tokens
   • Result: nassimjp/MiniCPM5-2B-Pashto

2. Stage 2: Supervised Fine-Tuning 🎯

   • Fine-tuned the surgically-enhanced model on Pashto instruction data
   • Used LoRA/QLoRA adapter techniques with very low learning rate (5e-6)
   • Fully merged into 16-bit (bfloat16) safetensors format

📋 Model Overview

🎯 Why This Model Exists

The Problem:

• ❌ Pashto is a low-resource language with limited AI support
• ❌ Original MiniCPM5-2B tokenizer splits many Pashto characters into multiple tokens
• ❌ Inefficient tokenization → poor instruction following
• ❌ No open-weight Pashto instruction-tuned models available

The Solution (Two-Stage Approach):

1. ✅ Stage 1: Tokenizer surgery — added 46 missing Pashto/Urdu atoms as single tokens
2. ✅ Stage 2: Fine-tuned on < 10K high-quality Pashto instruction pairs with conservative LR (5e-6)
3. ✅ Supports long-form generation (up to 40K tokens per sample)
4. ✅ Preserves Pashto script integrity
5. ✅ Openly available for the community

📊 Stage 1: Tokenizer Surgery Report

✅ Existing Single-Token Atoms (20)

⚠️ Split Atoms — FIXED via Surgery (46)

🔬 Embedding Forensics (After Surgery)

✅ No exact duplicate embeddings detected

📊 Stage 2: Training Highlights & Loss Metrics

The fine-tuning run was executed locally using an NVIDIA RTX 4070 Ti SUPER (16GB VRAM) across 8,952 steps.

Key Training Parameters

Loss Metrics

Why 5e-6 Learning Rate?

Using a very low learning rate (5e-6) was a deliberate choice:

• ✅ Preserves surgically-added token embeddings — prevents catastrophic forgetting of the 46 new tokens
• ✅ Conservative adaptation — the base model already has strong language capabilities; we only need to teach it to follow instructions
• ✅ Prevents overfitting — with < 10K samples, aggressive LR would cause overfitting
• ✅ Stable gradient norms — maintained 1.4-2.2 throughout training
• ✅ Excellent convergence — loss dropped from 4.61 to 0.54 without instability

🚀 Quickstart & Usage

1. Using Transformers (Python)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Example 1: Pashto Question
prompt = "د پښتو ژبې او مصنوعي ځیرکتیا پرمختګ څه اهمیت لري؟"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Example 2: Check that Pashto atoms are single tokens
test_chars = "ښڅځڼږډټړېۍګ"
for char in test_chars:
    ids = tokenizer.encode(char, add_special_tokens=False)
    print(f"'{char}' → {ids}")  # All should be single tokens!

2. Long-Form Generation (up to 40K tokens)

long_prompt = "د افغانستان تاریخ..."  # Can be up to 40K tokens
inputs = tokenizer(long_prompt, return_tensors="pt", truncation=True, max_length=40000)

3. Streaming Generation

from transformers import TextStreamer

streamer = TextStreamer(tokenizer, skip_prompt=True)
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    streamer=streamer
)

4. Batch Inference

prompts = [
    "پښتو ژبه څنګه زده کړم؟",
    "د افغانستان پلازمینه څه ده؟",
    "پښتو ادب په اړه وږغاړئ"
]

inputs = tokenizer(prompts, padding=True, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, temperature=0.7)

5. Running via llama.cpp / GGUF

./llama-cli -m ./MiniCPM5-2B-SFT-Pashto-Instruct.F16.gguf \
  -p "د پښتو ژبې او مصنوعي ځیرکتیا پرمختګ څه اهمیت لري؟" \
  -n 256 \
  -t 8

⚙️ Training Details

Dataset

• Size: < 10,000 Pashto instruction-response pairs
• Format: Instruction + Response pairs
• Language: Pashto (ps) with some English (en) examples
• Max Length: Up to 40K tokens per sample

Training Configuration

Hardware

• GPU: NVIDIA RTX 4070 Ti SUPER
• VRAM: 16GB
• Training Duration: Several days (8,952 steps)

📂 Model Files

MiniCPM5-2B-SFT-Pashto-Instruct/
├── config.json                      ~1 KB
├── generation_config.json           ~1 KB
├── model.safetensors                ~4-5 GB
├── tokenizer.json                   ~9 MB (surgically enhanced)
├── tokenizer_config.json            ~1 KB
├── special_tokens_map.json          ~1 KB
└── training_logs.json               Training metrics

⚠️ Important Notes

🧪 Testing Results

All 66 Pashto/Urdu atoms are now single tokens:

🔗 Related Resources

• Base Model: openbmb/MiniCPM5-2B
• Tokenizer Surgery Model: nassimjp/MiniCPM5-2B-Pashto
• Original Paper: MiniCPM
• GitHub: OpenBMB/MiniCPM

📝 Citation

If you use this model, please cite:

@misc{minicpm5-pashto-instruct,
  author = {Nassimjp},
  title = {MiniCPM5-2B-SFT-Pashto-Instruct: Pashto Instruction-Tuned Model with Tokenizer Surgery},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct}}
}

@article{minicpm2024,
  title={MiniCPM: Unveiling the Potential of Small Language Models},
  author={Hu, Shengding and Ding, Ning and others},
  journal={arXiv preprint arXiv:2404.06395},
  year={2024}
}

📄 License

This model is released under the Apache License 2.0.

🤝 Acknowledgements

• OpenBMB for developing MiniCPM5-2B
• Hugging Face for the transformers library and model hosting
• iPashto.ai community for dataset contributions and testing

🌐 Project Context

Developed under the iPashto.ai framework dedicated to advancing Pashto Natural Language Processing (NLP), specialized datasets, and open-weight AI accessibility for low-resource languages.

📬 Contact & Support

• Model Page: nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct
• Issues: Please open an issue on the model page
• Suggestions: Feedback welcome!

Created with ❤️ for the Pashto community

---