Gemma 4 E2B — Arabic + English Vision (Pruned)
A pruned version of google/gemma-4-e2b-it optimized for Arabic and English vision-language tasks. Unnecessary language tokens and the audio encoder have been removed, resulting in a smaller and more efficient model while retaining full vision and text generation capabilities.
What was done
| Component | Original | Pruned |
|---|---|---|
| Vocabulary | 262,144 tokens | 209,836 tokens (Arabic + English + special) |
| Vision encoder | ✅ 16 layers | ✅ KEPT |
| Audio encoder | ✅ 12 layers | ❌ Removed |
| Decoder layers | 35 layers | 35 layers (untouched) |
| PLE tables | 262,144 × 256 × 35 | 209,836 × 256 × 35 |
| Total size (bf16) | 10.2 GB | 8.5 GB |
| Size reduction | — | 17% |
Token breakdown
- Arabic tokens kept: 8,460
- English/Latin tokens kept: 201,096
- Special tokens: 24
- Byte fallbacks: 256
- Total kept: 209,836 (80.0% of original)
- Dropped: 52,308 (CJK, Cyrillic, Devanagari, Thai, etc.)
Why this exists
Gemma 4's Per-Layer Embedding (PLE) architecture stores a separate embedding table for each of its 35 decoder layers. With 262K vocab, that's 4.7 GB of PLE tables alone (53% of the model). By pruning to Arabic+English only, we cut this to ~3.8 GB.
The vision encoder is kept intact for image understanding tasks.
Benchmark: Arabic text recognition (zero-shot, no fine-tuning)
Evaluated on loay/ar_stage1_probe — 100 samples, seed=42:
| Model | Params | VRAM | CER ↓ | Exact Match |
|---|---|---|---|---|
| This model (pruned) | 4.25B | 8.5 GB | 0.1168 | 23/100 |
| google/gemma-4-e2b-it (original) | 5.10B | 10.2 GB | 0.1168 | 23/100 |
Identical output on all 100 samples — pruning is lossless for Arabic+English tasks.
Intended use
- Arabic + English vision-language tasks
- Document understanding
- Fine-tuning base for bilingual Arabic/English applications
- Any task where CJK/Cyrillic/Indic language support is not needed
Usage
from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained(
"ml-agent-explorers/gemma-4-e2b-arabic-english-vision",
dtype="bfloat16",
device_map="auto",
)
processor = AutoProcessor.from_pretrained(
"ml-agent-explorers/gemma-4-e2b-arabic-english-vision"
)