Fasih-2B (فصيح)
A 2.09 billion parameter Arabic language model trained from scratch using the nanochat framework.
Model Details
| Property | Value |
|---|---|
| Parameters | 2,088,768,048 (2.09B) |
| Architecture | NanoChatGPT (Transformer decoder) |
| Layers | 24 |
| Hidden size | 1536 |
| Attention heads | 12 |
| Vocabulary | 65,536 (Arabic BPE) |
| Context length | 2,048 tokens |
| Precision | bfloat16 |
| Language | Arabic |
Training
Pretraining
- Dataset: AraMix (8.19B tokens, Chinchilla-optimal)
- Steps: 7,812 steps
- Hardware: 8x NVIDIA H20 GPUs
- Batch size: 1,048,576 tokens
SFT (Supervised Fine-Tuning)
- Datasets: alpaca-gpt4-arabic (50K) + CIDAR (10K) + ArabicMMLU (x3) + ArabicGSM8K (x4) + Fasih identity conversations
- Total mixture: 90,621 rows
- Steps: 16 (1 epoch)
Evaluation Results
| Benchmark | Accuracy | Random Baseline |
|---|---|---|
| ArabicMMLU | 29.91% | 25% |
| ACVA | 49.29% | 50% |
| ArabicGSM8K | 4.47% | 0% |
| ChatCORE | 0.0175 | 0.0 |
Usage
This model uses the nanochat framework. To use it:
git clone https://github.com/karpathy/nanochat
cd nanochat
# Copy model files to $NANOCHAT_BASE_DIR/chatsft_checkpoints/d24/
python -m scripts.chat_web -i sft
Chat Format
The model uses nanochat's special token format:
<|bos|><|user_start|>مرحبا، من أنت؟<|user_end|><|assistant_start|>
Limitations
- Small model (2B params) — limited knowledge and reasoning compared to larger models
- Trained primarily on Arabic text — limited multilingual capability
- Short context (2048 tokens)
- This is a research/educational model trained from scratch in ~12 hours
License
Apache 2.0