Joint training-operation checkpoint
This repository contains one trained member of a controlled five-run BabyLM Strict-Small family. The runs share source data, tokenizer, backbone, initialization seed, primary masked-language-model objective, main optimization schedule, and nominal exposure. There is one trained run per recipe, and added objectives require unequal training compute. Comparisons are descriptive.
Model and data
| Property | Value |
|---|---|
| Architecture | custom LTG-BERT-style masked encoder with relative attention |
| Parameters | 33,320,644 |
| Layers / hidden size | 12 / 384 |
| Attention heads / FFN size | 6 / 1,280 |
| Context length | 512 |
| Tokenizer | byte-level BPE, 16,384 entries |
| Primary objective | complementary whole-word masking + data2vec |
| Source corpus | 9,572,138 words |
| Deterministic replay | 427,862 words |
| Corpus per pass | 10,000,000 words |
| Nominal exposure | 100,000,000 words |
Training operations
This run combines two training-only operations:
- A complexity curriculum and rare-token-linked document-half recombination.
- A noisy causal auxiliary objective on one quarter of batch rows. The primary bidirectional masked LM keeps its complete context.
The operations change corpus presentation and add an auxiliary prediction loss. One trained run cannot identify an interaction, empirical variation-set learning, or a human memory mechanism.
Shared training settings include LAMB with peak learning rate 0.0035, a cosine schedule with 1.6% warmup and cooldown, bfloat16, 16,384 tokens per update, data2vec weight 0.5, and EMA decay 0.9998.
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "miguelcsx/mosaic-joint-d384-bbpe16k"
tokenizer = AutoTokenizer.from_pretrained(repo_id, revision="main")
model = AutoModelForMaskedLM.from_pretrained(
repo_id,
revision="main",
trust_remote_code=True,
)
main and chck_100M contain the final checkpoint. Earlier checkpoints are
stored as chck_1M through chck_10M, then chck_20M through chck_90M.
Remote code is required for the custom inference class. Review tolm.py
before loading it in a security-sensitive environment. The canonical code is
maintained at github.com/miguelcsx/tolm.
Evaluation and provenance
All required checkpoint revisions were evaluated with the same BabyLM
fast suite used for the controlled family. These outputs are local development
evidence, not a leaderboard rank or a superiority claim. Machine-readable
configuration and file provenance are recorded in training_manifest.json
and release_manifest.json.
Limitations
- This is one trained run, so it does not estimate training-run variance.
- The model is a small English masked LM, not a general-purpose assistant.
- The recipe label describes implemented operations. It does not establish child-like input or memory processing, or causal claims about children.