BabyLM 2026 Strict-Small Semantic Cloze Multi-Task MLM
This is the final checkpoint from the multitask_ewok_semantic_cloze_v1
experiment for BabyLM 2026 Strict-Small.
The model is a compact BERT-style masked language model trained from scratch on the
BabyLM 2026 Strict-Small corpus with a small set of auxiliary training tasks. The
main added objective is an MLM-head semantic cloze ranking task: each cloze example
contains a single [MASK], a good single-token completion, and a bad single-token
completion. The model scores the good and bad completions with the standard MLM head.
Intended Use
This checkpoint is intended for BabyLM 2026 shared-task evaluation and for reproducing the accompanying multi-task semantic cloze experiment. It is not intended as a general-purpose language model.
Training Setup
- Track: BabyLM 2026 Strict-Small
- Training data:
BabyLM-community/BabyLM-2026-Strict-Small - Tokenizer:
bert-base-uncased - Architecture: compact BERT masked language model
- Hidden size: 256
- Layers: 4
- Attention heads: 4
- Intermediate size: 1024
- Parameters: 11,201,338
- Maximum sequence length: 128
- Batch size: 96
- Maximum steps: 10,000
- Actual stopped step: 9,500
- Learning rate: 5e-4
- Weight decay: 0.01
- Warmup fraction: 0.06
- Precision: fp32
- Seed: 1
- Device used for training: NVIDIA RTX A6000
Early stopping monitored validation MLM loss with patience 2, minimum delta 0.01, and start-after step 9000.
Task Mixture
| Task | Sampling probability |
|---|---|
| MLM | 0.600 |
| Replaced-token detection | 0.100 |
| Connective prediction | 0.075 |
| Definiteness prediction | 0.075 |
| Collocation | 0.050 |
| Grammar minimal pair | 0.050 |
| Semantic cloze ranking | 0.050 |
The semantic cloze ranking loss is:
softplus(-(score_good - score_bad))
where score_good and score_bad are MLM log-probabilities at the masked position.
No classifier head or CLS pooling is used for this objective.
The semantic cloze examples are synthetic, single-mask examples over affordance, animate agency, part-whole, physical-property, and typical-location templates. No official EWoK or BLiMP evaluation items were used to generate the auxiliary data.
Official Fast Evaluation Snapshot
These local official fast-eval scores are from the experiment records in the submission repository.
| System | BLiMP | Supplement | EWoK | Entity | Eye | SPR |
|---|---|---|---|---|---|---|
| MLM baseline | 64.33 | 54.00 | 50.55 | 15.56 | 2.47 | 2.42 |
| BLiMP-pair repair best | 54.80 | 51.60 | 51.00 | 27.54 | 8.02 | 3.05 |
| Semantic cloze best | 54.10 | 57.60 | 50.73 | 36.61 | 7.56 | 3.11 |
| Semantic cloze final | 55.26 | 54.40 | 52.00 | 27.44 | 8.18 | 3.71 |
The final checkpoint is the recommended checkpoint for EWoK-focused reporting. It improves EWoK over the MLM baseline in this local comparison, but trades off against the best-validation checkpoint on Entity Tracking.
Files
model.safetensors: MLM-compatible model weightsconfig.json: BERT model configurationtokenizer.json,tokenizer_config.json: tokenizer filestraining_config.yaml: training configurationresults_summary.json: local experiment summary
Caveats
This is a single-seed research checkpoint. The result should be presented cautiously: semantic cloze produced the best EWoK score in the isolated experiment series, while weight sweeps and relational cloze follow-ups did not improve the EWoK result further. Full official leaderboard submission artifacts still need to be generated with the BabyLM evaluation pipeline.