arm-gemma-e4b
arm-gemma-e4b is Gemma-4-E4B adapted to Armenian by continued pretraining. To our knowledge, it is the first open Armenian LLM released together with its complete training corpus and recipe — every training token is either public (ArmWeb, ArmSTEM, FineWeb-Edu, Stack-smol) or reproducible from the released pipeline.
This is a base model: no instruction tuning, no chat template. Use it for Armenian text continuation, likelihood scoring, or as a starting point for Armenian SFT.
Recipe
- Base: Gemma-4-E4B, stock tokenizer (best Armenian fertility among CPT candidates: 4.15 tokens/word; vocabulary extension was sharply harmful in our ablation at this budget).
- Continued pretraining: 10B tokens, sequence-packed at length 4096,
LR 3×10⁻⁵ cosine. The architectural context window is unchanged from the
base (128K,
max_position_embeddings=131072); CPT only exercised positions up to 4096, and long-context behavior beyond that is inherited from the base and not specifically evaluated. - Mixture (epoch-capped): 69% ArmWeb, 4% ArmSTEM-HY, 2% ArmSTEM-EN,
20% English web replay (FineWeb-Edu), 5% code (Stack-smol) — every ArmWeb
token seen ~2× and every ArmSTEM token 7–9× (Gemma-tokenizer accounting).
The ArmSTEM stream is a 109,885-item subset of the corpus (~30%, sampled
at random with balanced stratification across the math and science
pools); its manifest ships with ArmSTEM (
stats/cpt_training_subset_ids.txt, 104,630 items verbatim in the release, 5,255 superseded before release). A control run trained on the full 373K corpus (~1.3 epochs instead of 7–9) scores within noise of the released model (paper §5), so the repetition is costless. - Details: global batch 512 sequences (~2.1M tokens/step), 4,770 steps, 100 warmup steps, AdamW (β₂=0.95), weight decay 0.1, bf16, seed 42.
The mixture is the headline finding of the accompanying paper: news-only CPT catastrophically forgets (−21.2pp Belebele at LR 10⁻⁴); a gentler LR recovers two-thirds; 6% verified translated STEM data reverses forgetting entirely, ending +2.2pp above the unadapted base while keeping the fluency gains.
Results
Six-task Armenian likelihood suite (accuracy; harness: lm-eval):
| Task | Gemma-4-E4B (base) | arm-gemma-e4b |
|---|---|---|
| Belebele-hye | 0.619 | 0.716 |
| INCLUDE-Armenian | 0.416 | 0.456 |
| m-MMLU-hy | 0.343 | 0.337 |
| ARC-hy | 0.227 | 0.229 |
| HellaSwag-hy | 0.266 | 0.262 |
| MultiBLiMP-hye | 0.989 | 0.992 |
| Mean | 0.477 | 0.499 |
This is the highest six-task mean among all open Armenian models we evaluated — the best prior models score 0.471 (ArmenianGPT-1.0-3B) and 0.436 (HyGPT-10b), both below the unadapted base.
ArmBench-LLM generative tasks (base-model-appropriate metrics; full results in the paper appendix):
| Task | base | arm-gemma-e4b |
|---|---|---|
| SynDARin (EM) | 0.04 | 0.92 |
| Hartak (EM) | 0.02 | 0.82 |
| DREAM (EM) | 0.48 | 0.84 |
| Belebele gen. (EM) | 0.66 | 0.90 |
| Scientific MCQA (EM) | 0.86 | 1.00* |
| MMLU-Pro-Hy | 0.154 | 0.251 |
| Topic (14-class) | 0.004 | 0.482 |
*Audited: zero shared 8-grams between the benchmark items and the ArmSTEM training corpus (paper, Appendix).
Honest negatives: POS tagging regresses under CPT (0.18→0.01); exam mathematics is flat for the released model (1.75 points), and the full-corpus control run reaching 2.75 suggests data diversity rather than difficulty is the binding factor; instruction-dependent ArmBench tasks (judged generation, BLEU QA) are low for all base-style models including this one — they measure formatting, and we defer them to an instruction-tuned variant.
Training data & contamination
Training data was decontaminated by 13-gram overlap against ten Armenian evaluation sets and all ArmBench items, on both the English and Armenian side for translated data. The two remaining reported benchmarks, m-MMLU-hy and ARC-hy, were scanned post hoc: zero of 4.31M ArmWeb training documents and 4 of 372,907 ArmSTEM pairs share any 13-gram with their items. Existing public Armenian corpora overlap these benchmarks at 7.9–17.4% (see the ArmWeb card).
Limitations
Inherits Gemma-4's biases and terms of use. News-domain-concentrated Armenian exposure. Machine-translated STEM data verified for answer preservation and language identity, not stylistic fluency (though a two-annotator native-speaker audit rated 299 of 300 sampled problems valid, Cohen's κ = 1.0). No safety tuning.
Citation
@article{arakelyan2026armweb,
title = {From Zero to Hero: An Open LLM Ecosystem for Armenian},
author = {Arakelyan, Erik and Avetisyan, Khatun and Davtyan, Meri and Grigoryan, Heghine and Khachatryan, Nane and Shahsuvaryan, Hayk and Sergoyan, Henrik and Martirosyan, Vahan},
year = {2026},
journal = {arXiv preprint arXiv:2609.03350}
}