AraNeoBERT v4
A 297M-parameter modern Arabic encoder (NeoBERT/ModernBERT-style) pretrained with
MLM for 220,000 steps (57.7B tokens, 1.5 epochs) on a much larger deduped AraMix
web backbone + dialect/OSIAN/news/books blend (15.5% dialect by mixture weight).
28 layers, hidden 768, 12 heads; alternating local/global attention (window 128,
global every 3 layers), QK-Norm, RoPE (dual theta 10k local / 160k global),
FlexAttention + sequence packing, StableAdamW, WSD schedule, EMA (0.999, exported
weights are the EMA shadow), three-phase length curriculum (1024 -> 4096).
64k SentencePiece Unigram tokenizer with web_canonical normalization baked into
the fast tokenizer.
Held-out AraMix MLM pseudo-perplexity: 11.37 @ seq 4096, mask 0.30 (step 197k, last point before the final low-mask decay phase); 5.46 @ mask 0.15 at the finished step 220k (decay-phase eval, not directly comparable to the 0.30 number above or to earlier runs' 0.30-mask figures).
Usage
This model uses custom modeling code — load with trust_remote_code=True:
from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("miweru/AraNeoBERT-v4")
model = AutoModelForMaskedLM.from_pretrained("miweru/AraNeoBERT-v4", trust_remote_code=True)
AutoModel, AutoModelForMaskedLM, AutoModelForSequenceClassification and
AutoModelForTokenClassification heads are registered.
Downstream benchmark (finetuned, identical recipe across models, 3 seeds: 13/17/23)
| Task | AraNeoBERT-v4 | AraBERT v2 | AraModernBERT | MARBERT |
|---|---|---|---|---|
| AJGT sentiment (macro-F1) | 0.9296 | 0.9147 | 0.9064 | 0.9444 |
| ArSarcasm sentiment (macro-F1) | 0.7431 | 0.7138 | 0.7057 | 0.7463 |
| ANERcorp NER (entity-F1) | 0.5585* | 0.7504 | 0.6917 | 0.7968 |
* One of the three NER finetuning seeds (seed 17) collapsed during finetuning (entity-F1 0.045, a finetuning-run instability, not a base-model signal); the other two seeds scored 0.808 and 0.823 — above AraBERT v2 and AraModernBERT, approaching MARBERT. The reported mean includes the collapsed run for transparency; treat the 2-seed clean mean (0.815) as the more representative number pending a rerun of seed 17.
Competitive with/ahead of AraBERT v2 and AraModernBERT on sentiment, essentially tied with MARBERT there; strong on NER once the one unstable finetuning seed is set aside.
Research artifact; private. Built with Prof. Mustafa Jarrar's group.