Recombined curriculum checkpoint
This repository contains one trained member of a controlled five-run BabyLM Strict-Small family. The runs share source data, tokenizer, backbone, initialization seed, primary masked-language-model objective, main optimization schedule, and nominal exposure. There is one trained run per recipe, and added objectives require unequal training compute. Comparisons are descriptive.
Model and data
| Property | Value |
|---|---|
| Architecture | custom LTG-BERT-style masked encoder with relative attention |
| Parameters | 33,320,644 |
| Layers / hidden size | 12 / 384 |
| Attention heads / FFN size | 6 / 1,280 |
| Context length | 512 |
| Tokenizer | byte-level BPE, 16,384 entries |
| Primary objective | complementary whole-word masking + data2vec |
| Source corpus | 9,572,138 words |
| Deterministic replay | 427,862 words |
| Corpus per pass | 10,000,000 words |
| Nominal exposure | 100,000,000 words |
Training operation
Documents are ordered by subword tokens per word. The eligible fraction grows from 30% by 35 percentage points per epoch. On 40% of document visits, the first half is joined to the second half of a neighboring document that shares one of the source document's rare eligible tokens.
This is a corpus-presentation intervention. It is not an empirical variation-set manipulation and does not establish an effect on child language learning.
Shared training settings include LAMB with peak learning rate 0.0035, a cosine schedule with 1.6% warmup and cooldown, bfloat16, 16,384 tokens per update, data2vec weight 0.5, and EMA decay 0.9998.
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "miguelcsx/mosaic-views-d384-bbpe16k"
tokenizer = AutoTokenizer.from_pretrained(repo_id, revision="main")
model = AutoModelForMaskedLM.from_pretrained(
repo_id,
revision="main",
trust_remote_code=True,
)
main contains the final 100M-word checkpoint. Earlier checkpoints are stored
as chck_1M through chck_10M, then chck_20M through chck_90M. This
repository has no chck_100M branch.
Remote code is required for the custom inference class. Review tolm.py
before loading it in a security-sensitive environment. The canonical code is
maintained at github.com/miguelcsx/tolm.
Evaluation and provenance
The required checkpoint revisions were evaluated with the same BabyLM fast suite
used for the controlled family. These outputs are local development evidence,
not a leaderboard rank or a superiority claim. Machine-readable
configuration and file provenance are recorded in training_manifest.json
and release_manifest.json.
Limitations
- This is one trained run, so it does not estimate training-run variance.
- The model is a small English masked LM, not a general-purpose assistant.
- The recipe label describes an implemented operation. It does not establish child-like input processing or a causal claim about children.