Historical teacher checkpoint
This repository contains one trained member of a controlled five-run BabyLM Strict-Small family. The runs share source data, tokenizer, backbone, initialization seed, primary masked-language-model objective, main optimization schedule, and nominal exposure. There is one trained run per recipe, and added objectives require unequal training compute. Comparisons are descriptive.
Model and data
| Property | Value |
|---|---|
| Architecture | custom LTG-BERT-style masked encoder with relative attention |
| Parameters | 33,320,644 |
| Layers / hidden size | 12 / 384 |
| Attention heads / FFN size | 6 / 1,280 |
| Context length | 512 |
| Tokenizer | byte-level BPE, 16,384 entries |
| Primary objective | complementary whole-word masking + data2vec |
| Source corpus | 9,572,138 words |
| Deterministic replay | 427,862 words |
| Corpus per pass | 10,000,000 words |
| Nominal exposure | 100,000,000 words |
Training operations
This run uses the joint corpus-recombination and noisy-auxiliary recipe. It also stores its own 40M-word EMA parameters. After 40M, the stored teacher is evaluated on the same masked targets. Up to 25% of targets receive a temperature-2 KL loss of weight 0.20 when teacher cross-entropy is lower than student cross-entropy by more than 0.05.
This is selective self-distillation from an earlier state of the same run. A comparison with a separately trained checkpoint does not identify consolidation, preservation, or forgetting effects.
Shared training settings include LAMB with peak learning rate 0.0035, a cosine schedule with 1.6% warmup and cooldown, bfloat16, 16,384 tokens per update, data2vec weight 0.5, and EMA decay 0.9998.
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "miguelcsx/mosaic-sleep-d384-bbpe16k"
tokenizer = AutoTokenizer.from_pretrained(repo_id, revision="main")
model = AutoModelForMaskedLM.from_pretrained(
repo_id,
revision="main",
trust_remote_code=True,
)
main and chck_100M contain the final checkpoint. Earlier checkpoints are
stored as chck_1M through chck_10M, then chck_20M through chck_90M.
Remote code is required for the custom inference class. Review tolm.py
before loading it in a security-sensitive environment. The canonical code is
maintained at github.com/miguelcsx/tolm.
Local evaluation record
The final checkpoint completed local runs of the BabyLM zero-shot, reading, AoA, and fine-tuning pipelines. These are development results; this card makes no leaderboard rank or superiority claim.
| Evaluation | Score |
|---|---|
| BLiMP | 68.86 |
| BLiMP Supplement | 58.81 |
| EWoK | 51.61 |
| Entity Tracking | 22.84 |
| COMPS | 53.16 |
| GlobalPIQA | 37.62 |
| Eye-tracking | 10.89 |
| Self-paced reading | 4.81 |
| Reading composite | 7.85 |
| AoA, model-specific support | -0.1309 |
Fine-tuning development results are BoolQ 66.30, MNLI 58.50, MRPC F1 84.49, MultiRC 68.56, QQP F1 72.36, RTE 69.06, and WSC 65.38.
The AoA value is the output of the official BabyLM M-NTP pipeline on this model's eligible words. It is an estimator-specific correspondence measure, not a literal acquisition age or a recipe effect. This family member was used for the Strict-Small submission before the later four-run post-hoc AoA comparison; AoA played no role in submission or checkpoint selection.
Machine-readable configuration and file provenance are recorded in
training_manifest.json and release_manifest.json.
Limitations
- This is one trained run, so it does not estimate training-run variance.
- The model is a small English masked LM, not a general-purpose assistant.
- The recipe label describes implemented training operations. It does not establish consolidation, child-like memory, or causal claims about children.