BabySteps MurphysLaw 10M (mixed)
BabyLM 2026 Challenge — strict-small track submission (Team BabySteps)
A GPT-BERT hybrid masked/causal language model trained on the official BabyLM 2026 strict-small corpus (10M words), using the AdaMuon optimizer with a tuned learning rate and tail-weight averaging over the final 20% of training.
Our central finding: after data-centric experimentation (curriculum-ordered corpora, story and textbook corpora, synthetic data generation, teacher-supervised objectives), none of those interventions beat a faithful GPT-BERT reconstruction trained on the official corpus with a Muon-family optimizer and a careful learning-rate sweep.
Model details
| Architecture | GPT-BERT (hybrid masked + causal objective, single transformer stack) |
| Parameters | 39,337,540 |
| Hidden size | 384 |
| Layers | 12 |
| Attention heads | 6 |
| Intermediate size | 1280 |
| Vocabulary | 16,384 (BabyLM GPT-BERT baseline BPE) |
| Training sequence length | 128 (architecture supports 512) |
| Position embeddings | relative, bucket size 32 |
| Precision | bf16 |
Training
| Corpus | Official BabyLM 2026 strict-small (10M words), unmodified |
| Objective | mixed (causal_per_masked = 1, masked_per_causal = 0) |
| Optimizer | AdaMuon (matrix params) + AdamW-style auxiliary (embeddings, norms, head) |
| Learning rate | 0.02 (matrix) / 0.006 (auxiliary) |
| Schedule | linear warmup (1.6%) + two-phase decay |
| Weight decay | 0.1 · grad clip 2.0 · z-loss 1e-4 |
| Steps × tokens/step | 9,914 × 16,384 (≈10 epochs) |
| Seed | 42 |
| Final weights | tail average over the last 20% of training steps |
| Hardware | 1 × NVIDIA RTX 5090, ~30 mins training time |
Tail averaging left zero-shot scores essentially unchanged (±0.3 BLiMP) but improved finetuned SuperGLUE by ≈1.6 points on the development twin of this configuration.
Checkpoints
Intermediate checkpoints are published as branches following the challenge's naming convention, covering the strict-small milestone schedule (1M–10M words in 1M steps, then 10M–100M in 10M steps — 19 checkpoints total):
from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained(
"svsatheesh/BabySteps_MurphysLaw-10M-mixed",
revision="chck_7M", # any of chck_1M ... chck_100M
trust_remote_code=True,
)
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("svsatheesh/BabySteps_MurphysLaw-10M-mixed")
model = AutoModelForMaskedLM.from_pretrained(
"svsatheesh/BabySteps_MurphysLaw-10M-mixed", trust_remote_code=True
)
trust_remote_code=True is required — the GPT-BERT modeling code ships with the repo
(modeling_gpt_bert.py, configuration_gpt_bert.py). The model can be loaded as
AutoModelForMaskedLM or AutoModelForCausalLM; all reported results use MNTP
(masked-next-token-prediction) evaluation, which outperformed pure-causal and pure-masked
evaluation on the aggregate in our sweeps.
Results (official BabyLM 2026 evaluation pipeline, MNTP)
| Task | Score |
|---|---|
| BLiMP | 71.59 |
| BLiMP Supplement | 63.93 |
| EWoK | 51.94 |
| Entity Tracking | 27.95 |
| COMPS | ≈54.3 |
| Global PIQA | ≈35 |
| SuperGLUE (finetuned avg) | ≈69.9 |
| AoA correlation (CDI) | −0.151 |
SuperGLUE finetuning used the official pipeline at learning rate 8–9e-5, seed 44. SuperGLUE convention: F1 for MRPC and QQP, accuracy elsewhere.
Training data
Only the official BabyLM 2026 strict-small corpus was used — no custom data, no synthetic data augmentation, no teacher models, no external pretraining. The model is fully compliant with the strict-small track's data budget.
Limitations
This is a deliberately small model trained on a developmentally plausible data budget. It is a research artifact for studying sample-efficient pretraining, not a general-purpose language model: it has limited world knowledge, produces low-quality free-form generation, and inherits whatever biases exist in the BabyLM corpus (CHILDES, Project Gutenberg, OpenSubtitles, Simple Wikipedia, BNC, Switchboard). It should not be deployed in any user-facing application.
Reported scores are from a single training run.
Citation
@inproceedings{satheesh2026babysteps,
title = {BabySteps at BabyLM 2026: Optimizer Choice and Hyperparameter Tuning
Beat Data Interventions for Sample-Efficient GPT-BERT Pretraining},
author = {Satheesh, Siddhi; Murillo, Jorge; Chen, Yubei},
booktitle = {Proceedings of the BabyLM Challenge 2026},
year = {2026}
}
Built on GPT-BERT (Charpentier & Samuel, 2024), Muon (Jordan et al., 2024), and AdaMuon (Si et al., 2025).