Chatty BERT — BabyLM 2026 Strict

Name clarification: despite its BabyLM submission name, Chatty BERT is a decoder-only Llama-style causal language model. It is not a BERT-family model and it is not instruction-tuned.

Chatty BERT is a 127.6M-parameter language model trained from scratch for the BabyLM 2026 Strict track. It was developed as a strong causal baseline for data-constrained pretraining. The selected system combines a compact Llama architecture, Muon optimization, and Liger kernels.

The model is intended as a research artifact. Its strongest measured behavior is on grammatical minimal-pair evaluation. It remains weak on tasks requiring online state updates, and it should not be treated as a general-purpose chat or reasoning model.

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = (
    "Jorgeis1/"
    "babylm-2026-strict-llama-12l896-muon-auxlr1e4-bos-seed42-20260720"
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

This is a raw causal language model. Use ordinary text prompts rather than a chat template.

Model details

Training data

The model was trained on the English BabyLM 2026 Strict corpus, which contains 100 million words. The same fixed corpus was processed for 10 epochs. The training recipe did not add the project's experimental entity-state curricula, COMPS curriculum, spatial-ranking objective, or NextLat objective.

The 100M-word limit describes the corpus size, not the cumulative number of token presentations. Byte-level BPE tokenization produced approximately 177M input tokens per epoch.

Training procedure

Muon was applied to eligible hidden matrix parameters. Embeddings, norms, biases, and other auxiliary parameters used separate AdamW-style parameter groups.

The exact reported-score recipe is available in the BabySteps repository.

Evaluation

The model was evaluated with the pinned official BabyLM 2026 evaluation pipeline. Zero-shot tasks use causal likelihood comparisons. SuperGLUE tasks were evaluated after task-specific fine-tuning.

Scores are percentages unless otherwise noted.

Global-PIQA was 25.24 on the parallel split and 54.00 on the nonparallel split. The SuperGLUE macro-primary score uses accuracy for BoolQ, MNLI, MultiRC, RTE, and WSC, and F1 for MRPC and QQP.

Comparison with the reevaluated GPT-2 Strict baseline

This is a comparison of practical systems, not a controlled architecture ablation. Architecture, optimizer, kernels, implementation, and hyperparameters differ between the systems.

Entity-tracking findings

Entity Tracking tests whether a model can maintain a small world state while objects are added, removed, or moved between locations. Each official example has five candidate continuations, so nominal chance accuracy is 20%.

Chatty BERT scored 19.60%, approximately chance. On zero-operation cases, the three task-family scores ranged from 25.59% to 33.46%. With one or more state-changing operations, most family-by-operation scores were between about 14% and 20%. This pattern suggests that the model can sometimes retrieve an initial inventory but does not reliably compose subsequent state updates.

The score was produced with the pinned official evaluator included in the BabySteps repository.

Related curriculum experiments

The BabySteps project separately tested targeted entity-state curricula on 10M-word GPT-BERT models. Those curricula were not applied to Chatty BERT and should not be described as part of this checkpoint's training.

The principal findings were:

• An early container-state curriculum taught a fixed empty-answer phrase rather than general state tracking. Empty-answer accuracy rose to 79.01% and 93.37% at the 1% and 2% data doses, while non-empty accuracy remained at 19.40% and 18.57%.
• A measurement audit found that the historical summed-likelihood metric was strongly confounded by continuation length and the short answer nothing. Randomly initialized models obtained approximately 41% on that nominally 20%-chance legacy metric.
• A cleaner multidomain curriculum was tested in three matched GPT-BERT seeds. Replacing 1% of the corpus with this material reduced IID accuracy from 29.21% to 24.21% and did not produce a stable out-of-distribution gain.

The project-level conclusion is that ordinary exposure to these small synthetic curricula did not teach a robust, reusable state-update procedure. See the entity-tracking research summary for the complete measurement audit and experimental progression.

Next-latent prediction experiments

NextLat was an experimental training-only auxiliary objective that predicted a future hidden state. It was motivated by the possibility that explicit latent dynamics could encourage a model to represent how a sequence evolves rather than relying only on surface next-token associations.

NextLat is disabled in Chatty BERT. It is not part of this checkpoint's training recipe or inference architecture.

The completed single-seed Llama screenings did not establish a reproducible benefit:

Horizon 2 looked promising in the initial comparison but underperformed the baseline in a later matched recipe. The ordering also changed with the fine-tuning learning rate, and the repository does not contain a consolidated matched multi-seed downstream analysis.

The supported conclusion is therefore:

Next-latent prediction produced no demonstrated reproducible improvement across the tested training and fine-tuning recipes, so it was excluded from the released model.

See the auxiliary-objective summary for details.

Intended uses

Appropriate uses include:

• research on data-constrained pretraining;
• causal text continuation and likelihood scoring;
• linguistic, conceptual, and developmental evaluation;
• representation and learning-dynamics analysis;
• downstream fine-tuning experiments; and
• comparison with other BabyLM-scale models.

Out-of-scope uses

The model was not designed or validated for:

• deployment as a conversational assistant;
• high-stakes medical, legal, financial, or safety decisions;
• factual question answering without external verification;
• long-context processing;
• autonomous action or tool use; or
• moderation and other safety-critical classification.

Limitations

• This is a base model, not an instruction-following or safety-aligned model.
• The context window is limited to 1,024 tokens.
• Entity tracking is approximately at chance, especially after state-changing operations must be composed.
• COMPS and EWoK show only modest conceptual and world-knowledge performance.
• The model can generate false, incoherent, biased, or offensive text.
• No systematic toxicity, fairness, privacy, memorization, red-teaming, or adversarial-safety evaluation has been documented.
• The principal released result uses one pretraining seed. Small evaluation suites showed substantial run-to-run variation, including variation between nominally identical GPU retrains.
• Some evaluation suites contain artifacts or fuzzy labels. In particular, Entity Tracking required an answer-length audit, and some COMPS concept-property contrasts are debatable.
• The GPT-2 comparison changes multiple system components and should not be interpreted as proof of an isolated Llama architecture effect.

Reproducibility

Training and evaluation code, the exact training recipe, and the pinned BabyLM evaluation implementation are available in JorgeCMurillo/babylm-baby-steps.

Seed-identical GPU retrains are not guaranteed to be bit-identical. For an archival reproduction, record the exact PyTorch, CUDA, Transformers, Liger, GPU, driver, and evaluation-submodule versions alongside the run artifacts.

License

The project repository does not currently specify a software or model-weight license. The metadata value license: other reflects that unresolved status; it does not grant additional rights. A license should be selected and added before downstream redistribution or commercial use.

Citation

@misc{satheesh2026babysteps,
  title  = {BabySteps at BabyLM 2026: Stronger Optimization and Causal
            Baselines for Data-Constrained Pretraining},
  author = {Satheesh, Siddhi and Murillo, Jorge},
  year   = {2026}
}

Acknowledgments

This model was developed for the BabyLM Challenge using the official BabyLM 2026 data and evaluation infrastructure.