instanton-hybrid
A 40.2M-parameter GPT-BERT masked–causal hybrid trained entirely from scratch on the official BabyLM 2026 Strict-Small corpus (≤10M words).
Part of the instanton family — small models trained in minutes-to-hours,
submitted to the BabyLM 2026 Challenge.
| Parameters | 40,170,648 |
| Architecture | GPT-BERT (8 layers, 512 hidden, 8 heads, 1706 FFN) |
| Vocabulary | 16,384 (byte-level BPE, trained only on the in-budget corpus) |
| Training data | Official BabyLM 2026 Strict-Small, 9,972,806 words |
| Word exposure | 10 epochs ≈ 99.7M words (165.3M subword tokens) |
| Pretrained weights used | None — random init, per challenge rules |
Training objective
Following Charpentier & Samuel (2024), BERT or GPT: why not both?, the model is trained on a 50/50 blend of masked- and causal-LM objectives. Upstream splits the two objectives across DDP ranks; this model was trained on a single GPU, so the blend is reproduced in-process: half the micro-batches of each optimizer step come from a masked dataset and half from a causal dataset, gradients accumulate, then one step is taken.
Recipe
LAMB, max lr 0.007, cosine schedule with 1.6% warmup / 1.6% cooldown, weight decay 0.1, grad clip 2.0, z-loss 1e-4, mask ratio 0.30→0.15, sequence length ramped 128→256→512 at 70%/90% of training, 16,384 tokens per step, seed 42, bf16. ~2 hours on one A100-80GB.
Checkpoints
Nineteen word-exposure revisions are published as git branches, as required by
the challenge: chck_1M … chck_9M, chck_10M, chck_20M … chck_100M,
plus main (final). Load any of them with revision=.
Usage
The repository ships a vendored modelling wrapper, so trust_remote_code=True
is required. It loads under transformers 4.51.x; 5.x is not supported.
from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("qyxu1994/instanton-hybrid", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
"qyxu1994/instanton-hybrid", trust_remote_code=True) # or revision="chck_10M"
AutoModelForCausalLM also works and is what the age-of-acquisition evaluation
uses. Everything else should be scored as a masked LM (pseudo-log-likelihood
/ mntp), which is how the numbers below were produced.
Evaluation
Official BabyLM 2026 pipeline, mntp backend for zero-shot / GLUE / reading and
causal for AoA.
| Task | Score |
|---|---|
| BLiMP | 72.13 |
| BLiMP supplement | 60.86 |
| EWoK | 50.15 |
| Entity tracking (filtered) | 19.87 |
| COMPS | 52.96 |
| Global-PIQA | 37.14 |
| (Super)GLUE | 67.58 |
| Reading | 6.35 |
| Age of acquisition | 0 (r = −0.11, p = 0.081) |
| Leaderboard NLP average | 51.53 |
| Leaderboard Overall | 40.78 |
For comparison, the best causal decoder in this family (27M-parameter GPT-2, same 10M-word budget) reaches BLiMP 67.29 and GLUE 63.01 — the hybrid objective is worth roughly +4.8 BLiMP and +4.6 GLUE.
Known limitation
Like every model in this family, instanton-hybrid correlates negatively
with children's age-of-acquisition ordering, so it scores 0 on that benchmark.
This is a property of the model rather than of the measurement: the checkpoint
sweep is complete (19/19, 152,095 surprisal observations) and tokenizer
fragmentation was ruled out (215 of 237 scored CDI words are single tokens, and
controlling for subword count leaves the correlation unchanged).
Intended use
A research artifact for studying sample-efficient language acquisition. It is small, trained on <10M words, and is not suitable for deployment or for generating text people will rely on.
Citation
Data mixture and recipe are described in the accompanying BabyLM 2026 report, Reweighting Child-Directed and Conversational Data for Sample-Efficient BabyLM Pretraining. The architecture is from Charpentier & Samuel (2024).