Ghanam-7B-Base-Pashto-v0.1

Model Description

Ghanam-7B-Base-Pashto-v0.1 is an experimental base model that extends the Mistral-7B-v0.1 vocabulary to include 47 Pashto/Arabic characters. This is the first stage of creating a Pashto-capable language model, where the tokenizer has been modified and the embedding layer has been expanded to support Pashto script.

Key Features

• Vocabulary Size: 32,010 tokens (original 32,000 + 47 Pashto characters)
• Quantization: 4-bit NF4 with double quantization for efficient inference
• Memory Efficient: ~4GB VRAM usage for inference
• Pashto Script Support: Now recognizes all Pashto-specific characters including ښ، ږ، څ، ډ، ړ، ځ، ګ، ۍ، ې

Pashto Characters Added

ا ب پ ت ث ج چ ح خ د ذ ر ز ژ س ش ص ض ط ظ ع غ ف ق ک ګ گ ل م ن ڼ و ه ی ډ ړ ځ څ ښ ږ ۍ ې ء آ أ ؤ ئ

⚠️ Important Note

This model does not yet understand Pashto language semantics. It can:

• ✅ Tokenize Pashto text correctly
• ✅ Generate Pashto characters (if prompted in Pashto)
• ❌ Understand Pashto meaning or context
• ❌ Respond meaningfully in Pashto

The model still generates English text as it hasn't been fine-tuned on Pashto data.

Model Details

Model Description

This model is the first step in creating a Pashto language model based on Mistral-7B. The vocabulary was expanded using the Stanford vocabulary expansion method (mean resizing), where new embeddings were initialized using the mean of existing embeddings to minimize disruption to the model's performance.

• Developed by: Nassim JP (Community Project)
• Model type: Causal Language Model (Decoder-only Transformer)
• Language(s): English (original) + Pashto script support (tokenization only)
• License: Apache 2.0
• Base Model: Mistral-7B-v0.1
• Quantization: 4-bit NF4 (BitsAndBytes)
• Vocabulary Expansion Method: Stanford mean resizing

Model Sources

• Repository: [Coming Soon]
• Base Model: mistralai/Mistral-7B-v0.1

Uses

Direct Use

You can use this model for:

• Pashto tokenization and text preprocessing
• Inference experiments with Pashto prompts (though responses will be in English)
• As a starting point for fine-tuning on Pashto datasets
• Research on vocabulary expansion for low-resource languages

Downstream Use

This model is intended to be fine-tuned on Pashto text data for:

• Pashto language modeling
• Pashto text generation
• Pashto translation tasks
• Pashto question answering

Out-of-Scope Use

• Do not use this model for production Pashto applications without fine-tuning
• Do not expect Pashto understanding or generation in Pashto
• Not suitable for tasks requiring Pashto semantic comprehension
• Not tested for bias, toxicity, or safety in Pashto context

Bias, Risks, and Limitations

Technical Limitations

1. No Pashto Understanding: The model has only expanded vocabulary but has not learned Pashto semantics
2. English-Only Knowledge: All pre-training knowledge remains in English
3. Potential Performance Degradation: Vocabulary expansion may slightly impact English performance
4. Untested on Pashto Tasks: No evaluation has been conducted on Pashto benchmarks

Recommendations

• Use this model only as a foundation for Pashto fine-tuning
• Validate carefully before using in any production environment
• Expect English responses when prompting in Pashto (until fine-tuned)
• Consider this a research checkpoint rather than a production model

How to Get Started with the Model

Loading the Model (4-bit)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# Model path (local or from Hub)
model_path = "nassimjp/Ghanam-7B-Base-Pashto-v0.1"

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    quantization_config=bnb_config,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Set padding token (optional)
tokenizer.pad_token = tokenizer.eos_token

# Test tokenization
pashto_text = "پښتونخوا کې ډېر ښکلي ښارونه او ځنګلونه شته"
tokens = tokenizer.encode(pashto_text)
print(f"Tokens: {tokens}")
print(f"Decoded: {tokenizer.decode(tokens)}")

Generating Text

# Prompt in Pashto (model will respond in English)
prompt = "سلام، څنګه یاست؟"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        temperature=0.7,
        do_sample=True,
        repetition_penalty=1.2
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Inspecting New Tokens

# Check if Pashto characters are recognized
pashto_chars = ["ښ", "ږ", "څ", "ډ", "ړ", "ځ", "ګ"]
for char in pashto_chars:
    token_id = tokenizer.encode(char)[0]
    token_str = tokenizer.decode([token_id])
    print(f"Character: {char} -> Token ID: {token_id} -> Decoded: {token_str}")

Training Details

Training Data

No training data was used for this model. The vocabulary was expanded without any fine-tuning on Pashto text. The model retains only the original Mistral-7B pre-training knowledge.

Training Procedure

Vocabulary Expansion Method

The model underwent a two-stage process:

Stage 1: Tokenizer Modification

• Added 47 Pashto/Arabic characters to the original tokenizer
• Preserved all original tokens (no replacements)
• New vocabulary size: 32,010

Stage 2: Embedding Resizing

• Used Stanford vocabulary expansion method (mean resizing)
• New embeddings initialized with the mean of existing embeddings
• Both lm_head and embedding layers were resized
• No gradient updates or training performed

Training Hyperparameters

• Quantization: 4-bit NF4 with double quantization
• Compute dtype: bfloat16
• Method: Vocabulary expansion only (no fine-tuning)
• Embedding Initialization: Mean resizing

Evaluation

Testing Data, Factors & Metrics

Testing Data

Initial testing was performed using:

• Pashto sample texts to verify tokenization
• English prompts to ensure base functionality
• Character-level inspection to confirm tokenizer additions

Metrics

• Vocabulary Size: Successfully expanded from 32,000 to 32,010
• Tokenization Accuracy: All 47 characters are properly tokenized
• Generation Quality: English generation preserved
• Memory Usage: ~4GB VRAM for inference

Results

Environmental Impact

Carbon emissions were minimal as this model:

• Did not require any training

• Only performed vocabulary expansion and quantization

• Used a single GPU for a few minutes

• Hardware Type: NVIDIA A100 (or similar)

• Hours used: < 1 hour

• Cloud Provider: N/A (local)

• Compute Region: N/A

• Carbon Emitted: Negligible

Technical Specifications

Model Architecture and Objective

• Architecture: Transformer decoder (Mistral-7B)
• Number of Layers: 32
• Hidden Size: 4096
• Attention Heads: 32
• Intermediate Size: 14336
• Vocabulary Size: 32,010 (expanded from 32,000)
• Positional Encoding: Rotary Position Embeddings (RoPE)

Compute Infrastructure

Hardware

• CPU for embedding expansion
• GPU for quantization and inference testing

Software

• Transformers: v4.31.0+
• BitsAndBytes: v0.41.0+
• PyTorch: v2.0.0+
• Accelerate: v0.20.0+

Model Card Authors

• Nassim JP - Vocabulary expansion and quantization implementation
• Original model: Mistral AI team

Model Card Contact

For questions or collaboration:

• Hugging Face: nassimjp

Acknowledgments

• Mistral AI for the base model
• Stanford NLP for the vocabulary expansion method
• Hugging Face for the transformers library
• BitsAndBytes team for 4-bit quantization

Glossary

• Vocabulary Expansion: Adding new tokens to a model's tokenizer
• Mean Resizing: Initializing new embeddings with the mean of existing embeddings
• 4-bit NF4: 4-bit Normal Float quantization method
• Fine-tuning: Training a pre-trained model on domain-specific data

Next Steps

To make this model truly Pashto-capable, the next steps include:

1. Collect Pashto corpus (books, articles, web text)
2. Pre-train on Pashto data (continued pre-training)
3. Fine-tune for specific tasks (translation, QA, generation)
4. Evaluate on Pashto benchmarks
5. Deploy in production environments

Citation

If you use this model or the vocabulary expansion method, please cite:

@misc{ghanam2024pashto,
  author = {Nassim JP},
  title = {Ghanam-7B-Base-Pashto: Vocabulary Expansion for Pashto Language},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/nassimjp/Ghanam-7B-Base-Pashto-v0.1}}
}

Original Mistral-7B Citation

@article{jiang2023mistral,
  title={Mistral 7B},
  author={Jiang, Albert Q and others},
  journal={arXiv preprint arXiv:2310.06825},
  year={2023}
}

Additional Information

Usage Tips

1. Tokenization: The tokenizer now correctly handles all Pashto characters
2. Memory: 4-bit quantization allows running on consumer GPUs with 6-8GB VRAM
3. Generation: Use do_sample=True and temperature=0.7 for diverse outputs
4. Pad Token: If using batch generation, set pad_token=tokenizer.eos_token

Common Issues

Model Status: 🔬 Research Phase - Pre-fine-tuning checkpoint

Last Updated: June 2026