🧬 MiniCPM5-1B-Pashto

Pashto + Urdu Tokenizer Surgery Model

📋 Model Overview

This is a tokenizer-modified version of openbmb/MiniCPM5-1B with 46 Pashto and Urdu-specific characters added as single tokens to the vocabulary.

🔍 Why This Model Exists

MiniCPM5-1B's tokenizer does not natively support many Pashto and Urdu characters as single tokens. This causes:

• ❌ Characters being split into multiple tokens
• ❌ Inefficient encoding
• ❌ Poor representation for Pashto/Urdu text

This model fixes that by surgically adding 46 missing atoms as independent tokens.

📊 Complete Audit & Modification Report

✅ Existing Single-Token Atoms (20)

⚠️ Split Atoms — FIXED via Surgery (46)

🔬 Embedding Forensics

Model Configuration

{
    "architectures": ["LlamaForCausalLM"],
    "vocab_size": 130606,
    "hidden_size": 1536,
    "num_hidden_layers": 24,
    "max_position_embeddings": 131072,
    "tie_word_embeddings": False,
    "initializer_range": 0.02,
}

Input Embedding Statistics (New Atoms)

Most Similar Pair (New Atoms)

✅ No exact duplicate embeddings detected

Sample New Token Embedding Fingerprints

🚀 Usage

Installation

pip install transformers torch accelerate

Basic Loading

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model and tokenizer
model_id = "nassimjp/MiniCPM5-1B-Pashto"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

Tokenization Check

# Test Pashto characters
pashto_text = "سلام ښه راغلئ"

# Tokenize
tokens = tokenizer.encode(pashto_text, add_special_tokens=False)
print(f"Token IDs: {tokens}")

# Decode back
decoded = tokenizer.decode(tokens)
print(f"Decoded: {decoded}")

# Check individual atoms
for char in "ښڅځڼږډټړېۍګ":
    ids = tokenizer.encode(char, add_special_tokens=False)
    print(f"'{char}' → {ids}")

Text Generation

import torch

# Simple generation
prompt = "ښه راغلئ"
inputs = tokenizer.encode(prompt, return_tensors="pt")

# Generate
with torch.no_grad():
    outputs = model.generate(
        inputs,
        max_new_tokens=50,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Batch Processing

texts = [
    "سلام",
    "ښه راغلئ",
    "پښتو ژبه",
    "ستاسو نوم څه دی؟"
]

# Tokenize batch
encodings = tokenizer(
    texts,
    padding=True,
    return_tensors="pt"
)

print(encodings.input_ids.shape)  # (4, sequence_length)

⚙️ Technical Details

Surgery Methodology

1. Audit Phase

   • Iterated through 66 Pashto/Urdu characters
   • Detected which characters split into multiple tokens
   • Identified 46 characters needing surgery

2. Token Addition

   • Used tokenizer.add_tokens() to add 46 missing atoms
   • Resized model embeddings with model.resize_token_embeddings()
   • New tokens assigned IDs: 130560 → 130605

3. Independent Initialization

   • Input embeddings initialized with normal distribution
   • Output embeddings initialized independently
   • Both use initializer_range = 0.02
   • Not tied — maintains separate input and output spaces

4. Verification

   • Tokenization gate confirmed all 66 atoms as single tokens
   • No exact duplicate embeddings detected
   • Embedding norms and covariances verified

⚠️ Important Notes

📦 Model Files

MiniCPM5-1B-Pashto/
├── chat_template.jinja             0.01 MB
├── config.json                     0.00 MB
├── generation_config.json          0.00 MB
├── ipashto_surgery_info.txt        0.00 MB
├── model-00001-of-00002.safetensors  1,899.42 MB
├── model-00002-of-00002.safetensors    162.02 MB
├── model.safetensors.index.json    0.02 MB
├── tokenizer.json                  9.44 MB
└── tokenizer_config.json           0.00 MB

Total Size: 2.02 GB

🔗 Related Resources

• Base Model: openbmb/MiniCPM5-1B
• Original Paper: MiniCPM: Unveiling the Potential of Small Language Models
• GitHub: OpenBMB/MiniCPM

📝 Citation

If you use this model, please cite the original MiniCPM paper:

@article{minicpm2024,
  title={MiniCPM: Unveiling the Potential of Small Language Models},
  author={Hu, Shengding and Ding, Ning and others},
  journal={arXiv preprint arXiv:2404.06395},
  year={2024}
}

📄 License

This model is released under the Apache License 2.0.

🤝 Acknowledgements

• OpenBMB for developing and releasing MiniCPM5-1B
• Hugging Face for the transformers library and model hosting
• Kaggle for providing the compute environment

🧪 Testing Results

All 66 Pashto/Urdu atoms successfully tokenize as single tokens:

📞 Contact & Support

• Model Page: nassimjp/MiniCPM5-1B-Pashto
• Issues: Please open an issue on the model page
• Suggestions: Feedback welcome!

Created with ❤️ for the Pashto and Urdu communities

---