Noisy auxiliary checkpoint
This repository contains one trained member of a controlled five-run BabyLM Strict-Small family. The runs share source data, tokenizer, backbone, initialization seed, primary masked-language-model objective, main optimization schedule, and nominal exposure. There is one trained run per recipe, and added objectives require unequal training compute. Comparisons are descriptive.
Model and data
| Property | Value |
|---|---|
| Architecture | custom LTG-BERT-style masked encoder with relative attention |
| Parameters | 33,320,644 |
| Layers / hidden size | 12 / 384 |
| Attention heads / FFN size | 6 / 1,280 |
| Context length | 512 |
| Tokenizer | byte-level BPE, 16,384 entries |
| Primary objective | complementary whole-word masking + data2vec |
| Source corpus | 9,572,138 words |
| Deterministic replay | 427,862 words |
| Corpus per pass | 10,000,000 words |
| Nominal exposure | 100,000,000 words |
Training operation
On one quarter of batch rows, an extra causal forward encodes the complete prefix. A learned scalar precision at each position controls Gaussian noise in the hidden state. The corrupted state predicts the next word-starting token, and a squared penalty keeps mean precision near 0.35. The auxiliary loss has weight 0.15. The primary bidirectional masked LM retains its complete context.
The implemented operation is a noisy causal auxiliary objective. The exported model has no bounded-memory architecture, and the run does not identify a human memory mechanism.
Shared training settings include LAMB with peak learning rate 0.0035, a cosine schedule with 1.6% warmup and cooldown, bfloat16, 16,384 tokens per update, data2vec weight 0.5, and EMA decay 0.9998.
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "miguelcsx/mosaic-memory-d384-bbpe16k"
tokenizer = AutoTokenizer.from_pretrained(repo_id, revision="main")
model = AutoModelForMaskedLM.from_pretrained(
repo_id,
revision="main",
trust_remote_code=True,
)
main and chck_100M contain the final checkpoint. Earlier checkpoints are
stored as chck_1M through chck_10M, then chck_20M through chck_90M.
Remote code is required for the custom inference class. Review tolm.py
before loading it in a security-sensitive environment. The canonical code is
maintained at github.com/miguelcsx/tolm.
Evaluation and provenance
All required checkpoint revisions were evaluated with the same BabyLM
fast suite used for the controlled family. These outputs are local development
evidence, not a leaderboard rank or a superiority claim. Machine-readable
configuration and file provenance are recorded in training_manifest.json
and release_manifest.json.
Limitations
- This is one trained run, so it does not estimate training-run variance.
- The model is a small English masked LM, not a general-purpose assistant.
- The recipe label describes an implemented operation. It does not establish child-like memory or a causal claim about children.