P7

Model Details

• Model name: P7
• Challenge: BabyLM 2026
• Track: Strict-Small
• Backend: causal
• Repository: moral1/babylm_strict_small_2026
• Final full-eval checkpoint: 80M (words_seen=80,010,749)
• Training checkpoints: 1M-100M cumulative words
• 100M branch: uses the strictly compliant final_compliant endpoint (words_seen=99,992,923)
• Evaluation pipeline commit SHA: 3d57ddc8c40ee795c0b5e41b3a20251a9457a593
• Final predictions JSON SHA256: 1e41f721edd432379a8d253dc4936e5a7ea4f003662d037fa9d8c927f77ceefe

Architecture

• Base architecture: GPTBERTForMaskedLM
• Model type: gpt-bert
• Parameter count: 132037444
• Hidden size: 768
• Layers: 12
• Attention heads: 12
• Vocabulary size: 16384

Tokenizer

• Tokenizer type/source: self_trained_strictsmall
• Tokenizer training words: 2000168
• Tokenizer files: tokenizer.json, tokenizer_config.json, special_tokens_map.json

Training

• Training dataset budget: BabyLM Strict-Small 10M
• Submitted full-eval checkpoint: 80M checkpoint selected from the training trajectory
• Random seed: 73
• Training objective: NEEDS_USER_CONFIRMATION: see babytutor_run_config.json
• Checkpoint trajectory branches: chck_1M through chck_9M, then chck_10M through chck_100M every 10M words

The originally overshooting 100,002,969-word checkpoint is not used as the official 100M branch. The chck_100M branch points to the strict compliant endpoint with 99,992,923 words seen.

Intended Use

This model is intended for research and BabyLM 2026 Strict-Small evaluation workflows.

Limitations

The model was trained under the BabyLM Strict-Small data budget and is not intended as a general-purpose production language model. Outputs may be unreliable, biased, or inappropriate, and the model should be evaluated in context before any downstream use.

Data and License

Training data follows the BabyLM 2026 Strict-Small challenge data constraints. Dataset licensing and attribution follow the BabyLM challenge materials. Model code and weights are provided for research and evaluation use.