ModernBERT-Small (Compositional + Shuffled-Frequency Residual, Control) — BabyLM 2026 Strict-Small
This model is a ModernBERT-Small masked language model (RoPE, GeGLU, alternating local/global
attention, 384 hidden size, 16 layers, 6 attention heads) trained from scratch on the
BabyLM 2026 Strict-Small 10M-word corpus, as part of an ablation study on
parameter-efficient token embedding layers for developmentally-plausible pretraining under the
BabyLM Challenge's strict-small compute and data budget.
Embedding design
Standard transformer token embedding tables scale as vocab_size x hidden_size, which for this
model's 30,522-token vocabulary and 384 hidden size would be an 11.7M-parameter dense lookup
table. This checkpoint's base representation is the same compositional byte-n-gram embedding
used across this ablation sweep: each vocabulary token is decomposed into its raw UTF-8 byte
sequence, every byte n-gram (n = 1 to 4) is extracted and hashed into one of 8,192 shared
buckets, and the token's representation is the mean of its buckets' embeddings (a
128-dimensional EmbeddingBag table, ~1.0M parameters) rather than a dedicated per-token row.
On top of that shared composed representation, this variant adds a frequency-gated per-token
residual -- a learned vocab_size x 128 embedding added to the composed vector, scaled per
token by frequency / (frequency + tau) (tau = 100) so common tokens lean more on their own
residual and rare tokens rely mostly on the composition. Critically, this checkpoint is the
shuffled-frequency control: the frequency values used to compute each token's gating weight
are randomly permuted across the vocabulary (fixed seed 42) before being applied, so a token's
residual is gated by an unrelated token's frequency rather than its own. This isolates whether
any benefit from the frequency-gated residual comes from the correct frequency signal, or
merely from having some per-token residual-plus-gating mechanism -- a null-effect control
against the true frequency-gated variant in this sweep.
This is one variant in a broader comparison of embedding-layer parameterizations (dense, linear and MLP-style factorization, tensor-train decomposition, deterministic Fourier expansion, and compositional/frequency-adaptive variants) evaluated under identical data, tokenizer, optimizer, and training budget, to isolate the effect of the embedding layer's parameterization on downstream BabyLM evaluation performance.
Training data
BabyLM 2026 Strict-Small corpus (~10M words), tokenized with a byte-level BPE tokenizer trained on the same corpus (vocab size 30,522). No external data, synthetic augmentation, or human annotation beyond the corpus as officially released.
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained(
"remg1997/modernbert-small-phase6-composition-frequency-shuffled-babylm2026",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
"remg1997/modernbert-small-phase6-composition-frequency-shuffled-babylm2026"
)
Each chck_{N}M branch of this repository corresponds to a BabyLM-Challenge compliance
checkpoint (one per N million words of training data seen); main points at the final,
fully-trained checkpoint.
Evaluation
Evaluated with the official babylm-eval harness
(BLiMP, EWoK, entity tracking, COMPS, Global PIQA, reading-time correlation, GLUE/SuperGLUE
fine-tuning, and Age-of-Acquisition word-surprisal correlation) under the strict-small track.