Qiushi-BabyLM-35M Strict-Small v3
Qiushi-BabyLM-35M-Strict-Small-v3 is a 35.46M-parameter English masked language model developed autonomously by Qiushi Engine for sample-efficient pretraining under the BabyLM Strict-Small setting.
The model combines compact-view reinvestment with zero-output residual adapters. Compact semantic views preserve useful parallel experience while returning word budget to diverse natural text. The adapters add a learnable low-rank residual pathway to each DeBERTa-v2 encoder layer while preserving the base function at initialization.
Model
| Property | Value |
|---|---|
| Architecture | DeBERTa-v2 MLM with residual bottleneck adapters |
| Parameters | 35,463,008 |
| Layers | 8 |
| Hidden size | 480 |
| Attention heads | 8 |
| FFN size | 1,920 |
| Adapter bottleneck | 128 |
| Adapter scale | 1.75 |
| Vocabulary | 16,384 byte-level BPE tokens |
| Sequence length | 256 tokens |
| Objective | 15% whole-word masking |
Training
The tokenizer was trained on the same 10,000,000-word corpus used for model pretraining. The model was initialized from random weights and the released checkpoint was selected at exactly 82,012,495 counted word exposures.
Training used AdamW, a peak learning rate of 0.001, 6% linear warmup followed by cosine decay, an average batch size of 65,536 tokens, and seed 43022. The corpus combines BabyLM-source material, counted compact semantic views generated with Qwen3.5-9B, and reinvested natural-text diversity. The student received text only; no teacher weights, hidden states, logits, or tokenizer were transferred.
Evaluation
Complete BabyLM 2026 Strict-Small official-compatible evaluation:
| Metric | Score |
|---|---|
| BLiMP | 68.8127 |
| BLiMP Supplement | 62.9290 |
| EWoK | 50.2520 |
| Entity Tracking | 28.3731 |
| COMPS | 52.4293 |
| (Super)GLUE | 69.7662 |
| GlobalPIQA | 36.3495 |
| Reading | 8.5705 |
| AoA | 0.0000 |
| Overall | 41.9425 |
The checkpoint was independently reproduced from the training corpus with an identical model SHA-256 and identical logits. The full prediction carrier also includes all required fast-checkpoint evaluations.
Usage
The adapter implementation is part of the model, so trusted repository code must be enabled:
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "leslie721007/Qiushi-BabyLM-35M-Strict-Small-v3"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForMaskedLM.from_pretrained(
repo_id,
trust_remote_code=True,
)
text = f"The child {tokenizer.mask_token} the book."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
This is a research masked language model rather than an instruction-tuned chat model.
Integrity
SHA256SUMS records the exact public artifact identities. Credentials, private
system code, local paths, internal traces, and operational logs are excluded.
See METHOD.md for the public method summary.