instanton-hybrid-dialogue
A 40.2M-parameter GPT-BERT masked–causal hybrid trained entirely from scratch on a dialogue-heavy remix of the BabyLM 2026 Strict corpus, within the Strict-Small budget (≤10M words).
Part of the instanton family, submitted to the
BabyLM 2026 Challenge. It is the mixture-variant
counterpart to qyxu1994/instanton-hybrid,
which uses the official mixture; architecture and recipe are identical, so the
pair isolates the effect of the data mixture.
| Parameters | 40,170,648 |
| Architecture | GPT-BERT (8 layers, 512 hidden, 8 heads, 1706 FFN) |
| Vocabulary | 16,384 (byte-level BPE, trained only on the in-budget corpus) |
| Training data | Dialogue-heavy remix, 9,999,167 words |
| Word exposure | 10 epochs ≈ 99.8M words |
| Pretrained weights used | None — random init, per challenge rules |
Training data
Resampled from the official detoxified BabyLM 2026 Strict (100M) pool — no external data — by drawing contiguous 100-line blocks without replacement (seeded) until per-source word targets were met. The mixture upweights spoken and conversational text (~6.2M words from OpenSubtitles, BNC, Switchboard and CHILDES), with children's books and simple encyclopedia making up the rest.
The research question: under a fixed 10M-word budget, does upweighting cognitively motivated, child-directed and conversational data improve sample efficiency?
Training objective and recipe
50/50 masked/causal blend after Charpentier & Samuel (2024), reproduced in-process on a single GPU (half the micro-batches of each optimizer step masked, half causal). LAMB, max lr 0.007, cosine with 1.6% warmup / 1.6% cooldown, weight decay 0.1, grad clip 2.0, z-loss 1e-4, mask ratio 0.30→0.15, sequence length 128→256→512, 16,384 tokens/step, seed 42, bf16. ~2 hours on one A100-80GB.
Checkpoints
Nineteen word-exposure revisions as git branches: chck_1M … chck_9M,
chck_10M, chck_20M … chck_100M, plus main. Load with revision=.
Usage
Requires trust_remote_code=True (vendored modelling wrapper) and
transformers 4.51.x; 5.x is not supported.
from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained(
"qyxu1994/instanton-hybrid-dialogue", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
"qyxu1994/instanton-hybrid-dialogue", trust_remote_code=True)
Score as a masked LM (pseudo-log-likelihood / mntp).
Evaluation
| Task | This model | instanton-hybrid (official mixture) |
|---|---|---|
| BLiMP | 70.76 | 72.13 |
| BLiMP supplement | 61.06 | 60.86 |
| EWoK | 52.90 | 50.15 |
| Entity tracking (filtered) | 21.93 | 19.87 |
| COMPS | 53.15 | 52.96 |
| Global-PIQA | 29.25 | 37.14 |
| (Super)GLUE | 67.64 | 67.58 |
| Reading | 7.14 | 6.35 |
| Age of acquisition | 0 (r = −0.16, p = 0.008) | 0 (r = −0.11) |
| Leaderboard NLP average | 50.96 | 51.53 |
| Leaderboard Overall | 40.43 | 40.78 |
The dialogue-heavy mixture wins most zero-shot columns, GLUE and reading, and notably recovers the EWoK and entity-tracking ground the official-mixture hybrid gives up — evidence that the architecture and the mixture compose rather than compete. It nonetheless has the lower leaderboard Overall, because of the Global-PIQA result below.
Known limitations
Global-PIQA is below chance and unexplained. This model scores 16.5% on the 4-choice parallel subset (chance 25%) and 42% on the 2-choice non-parallel subset (chance 50%) — it systematically prefers the wrong solution. This is not a scoring artifact: the same code reproduces the official leaderboard score exactly for a sibling model. Nor is it the mixture alone: a causal GPT-2 trained on the same dialogue-heavy corpus scores 38.12. It appears specific to this model and we have no explanation for it. Treat this model's physical-commonsense behaviour as unreliable.
Age of acquisition is significantly negative (r = −0.165, p = 0.008), the
most negative in the instanton family. Upweighting child-directed and
conversational text moved this metric in the wrong direction, which is a
negative result worth reporting: feeding a model more child-like text does not
make its acquisition trajectory more child-like.
Intended use
A research artifact for studying sample-efficient language acquisition and data mixture effects. Trained on <10M words; not suitable for deployment. See the Global-PIQA limitation above before using it for anything commonsense-related.
Citation
Described in the accompanying BabyLM 2026 report, Reweighting Child-Directed and Conversational Data for Sample-Efficient BabyLM Pretraining. Architecture from Charpentier & Samuel (2024).