AraNeoBERT v5 (250M) — research artifact, not a recommended checkpoint
Arabic NeoBERT encoder, 247.6M parameters in the encoder (296.9M during training, including an untied LM head). Trained 2026-08-03 to 2026-08-22 on 39.59B decontaminated tokens for 2.0 epochs.
Read this first: v5 performs worse than v4 on every axis we measured. It is published for reproducibility of a negative result. For downstream use, take v4.
What v5 changed, and what it cost
Four things changed at once relative to v4 — the Muon optimizer, an anchor-centered neighbor-prediction objective, a decontaminated corpus with Wikipedia/Shamela/Tunisian dialect added, and 1.97 instead of 1.48 epochs. The result was negative, and because four variables moved together the regression is not attributable to any one of them from this run alone.
| v4 | v5 (EMA) | |
|---|---|---|
| Downstream, 24 paired comparisons | baseline | −1.12pt (8/24 positive) |
| Held-out MLM ppl (Wikipedia) | 15.33 | 17.81 |
| Held-out MLM ppl (news) | 7.85 | 11.44 |
| Multi-token infill, span 3, exact | 31.0% | 3.7% |
The eval-loss gap opens by step 5,000 and stays flat at ~0.24 nats for the next 160,000 steps. It is a level shift present from the start, not a late failure.
The mechanism, identified
The objective's two "independently parameterised predictors" were one
predictor. The offset head is decoder(h + e_j), with e_j a
zero-initialised vector added to the hidden state. Measured on the finished
model, ||e_j|| is 1.3% of ||h|| and KL(decoder(h) || decoder(h+e_j)) is
2e-4 nats, against 0.14 nats between two genuinely adjacent positions. head_j
was numerically the plain MLM head for all 306,725 steps.
Growing a 768-vector against a hidden state of norm 165 is far more expensive
than bending the trunk, and the decoder is shared — so instead of learning
offset heads, the auxiliary loss trained the MLM head at position i to emit a
blend of x_i, x_{i±1}, x_{i±2}.
The damage is confined to the objective's window (neighbor_window=2). Masking
two positions d apart and measuring the divergence between their predictions:
| separation | v4 | v5 |
|---|---|---|
| 1 | 8.36 | 0.009 |
| 2 | 12.34 | 0.12 |
| 3 | 13.14 | 11.43 |
| 5 | 13.34 | 12.44 |
A discontinuity exactly at the window boundary. At separation ≥ 3, v5's top-1 (0.703) and gold-token probability (0.597) are indistinguishable from v4's (0.708 / 0.609) — v5 is not a generally worse model, it is broken only inside the window. Between positions one apart, its argmax agrees 94% of the time (v4: 1.4%): it emits the same token twice.
Muon is excluded directly: pilot_mlm and pilot_muon share a corpus and
differ only in optimizer, and Muon slightly reduces the flattening.
The fix — a real per-offset projection ahead of the shared decoder — is
implemented (--neighbor-head-kind projection) and untested at scale. No
model here uses it. Full diagnosis: docs/POSTMORTEM_v5.md in the repo.
Two further findings
The decay phase is not free. Annealing on a narrow high-quality subset
(aramix_v3_sd) costs 55–85% held-out perplexity on text outside that subset,
in both v4 and v5. The jump sits exactly at the data switch, not at the
RoPE-theta change one phase earlier. For v5 it also destroyed multi-token
infill: 24.7% exact at step 220,000, 3.7% at the end. v4 does not show that
collapse, so it is an interaction with v5's configuration, not a property of
annealing as such.
ckpt_step220000 is the better artifact on infill (24.7% vs 3.7% exact at
span 3) and was not benchmarked downstream. It is not published here.
Training
250M NeoBERT (hidden 768, 28 layers, 12 heads, local/global attention with
window 128 and a global layer every 3, QK-norm, dual RoPE theta 10000/160000),
Muon optimizer, mlm_neighbors objective with window 2 and consistency lambda
0.1, source-conditional whole-word masking on the dialect sources (33.2% of
documents), WSD schedule over 306,725 steps at 262,144 tokens/step, sequence
packing, bf16.
One OOM at step 222,900 during the context-extension phase, recovered from step
222,000 with neighbor_max_pairs halved from 4096 to 2048 for the remaining
27%. That change was measured not to affect infill (24.7% before, 24.5% after).
Decontamination
The training corpus was checked against the AJGT / ArSarcasm / ASTD / ANERcorp splits with two complementary scans — whole-document hashing over raw text and token-sequence search in the pretokenized data, which find disjoint classes of leakage. 297 documents containing benchmark items verbatim were removed and the result verified by rescanning. Items occurring in ≥10 corpus documents were kept and are listed by name in the report; they are formulaic religious phrasing, not benchmark content.
Residual limit: 852 benchmark items (3.2%) are shorter than 6 tokens and cannot
be checked, and aramix_v3_sd/aramix_v4_sd have no surviving raw text, so
only the token-level scan covers them.
Intended use
Reproducing the negative result above, and as the offset_emb arm of an
ablation against a real per-offset head. Not recommended for downstream Arabic
NLP; use v4.