babylm-2026-mtl-stage2
A 213M-parameter GPT-2-style causal language model for the multilingual track of the BabyLM 2026 Challenge (English, Dutch, Chinese), trained from scratch within the 100M English-equivalent-word budget.
This is the zero-shot-oriented variant. It is identical to
babylm-2026-mtl-bigckpt
except that part of the real text is replaced by higher-quality synthetic data,
which raises zero-shot at the cost of fine-tuning.
Model details
| Architecture | GPT-2-style decoder with multi-token prediction (MTP) |
| Parameters | 212,790,272 (213M) |
| Layers / heads / hidden | 14 / 16 / 1024 |
| Context length | 512 |
| Tokenizer | byte-level BPE, 32,000 vocab |
| Training objective | next-token + MTP auxiliary heads ($k{=}4$, ramped $1\to4$ over epochs; only the first head is used at inference) |
| Optimizer | AdamW, peak LR $4.5\times10^{-4}$, 10 epochs |
| Track | BabyLM 2026 multilingual |
| Languages | English, Dutch, Chinese |
Training data
Same as babylm-2026-mtl-bigckpt
(BabyBabelLM + Tatoeba/OPUS-100 parallel text + task-family-targeted synthetic
text from Qwen3.5-4B), plus higher-quality synthetic data (Qwen3.5-9B,
~14% of the corpus) replacing part of the real text to maximize zero-shot.
Published training data: 1ghub/babylm-2026-mtl-data
Note on budget. The track measures the budget in bytes: the 100M-word tier is 543.00 MB, and a corpus qualifies while its UTF-8 size divided by the language premium stays at or below that. We use the coefficients published in the organizers' table: English 1.0, Dutch 1.0516, and simplified Chinese 0.93597, the script BabyBabelLM uses. (The 0.9894 listed in the call for papers is the coefficient that table gives for traditional Chinese.) Under this rule the training corpus is 527.7 MB of English-equivalent bytes, 97.2% of the cap.
Results
Development-set scores (percentages):
| Metric | Score |
|---|---|
| Zero-shot (16-task mean, ZS16) | 58.40 (higher than bigckpt) |
| Fine-tuning (22-task mean, FT22) | 39.52 |
| ZS — English / Dutch / Chinese | 60.52 / 60.27 / 54.03 |
Relative to bigckpt, replacing real text with higher-quality synthetic data
increases zero-shot (ZS16 57.82 → 58.40) and decreases fine-tuning
(FT22 41.25 → 39.52). This model illustrates one pole of that trade-off.
Intended use and limitations
Research model for data-efficient multilingual language modeling under the BabyLM setting. Small model, developmentally plausible data budget, not intended for production use. Scores are on development sets and are subject to selection bias from evaluation reuse.
Teacher models
Synthetic training text was generated (not distilled) by Qwen3.5-4B and Qwen3.5-9B, approved teachers for the challenge. Generation prompts used task-level descriptions.
Citation
Companion paper: Language Allocation and Targeted Data Design for a Multilingual BabyLM (BabyLM 2026 Workshop, EMNLP 2026).