Qiushi-BabyLM-36M-Strict-Small-v4
Qiushi-BabyLM-36M-Strict-Small-v4 is a 36.46M-parameter masked language model developed by Qiushi Engine for BabyLM 2026 Strict-Small. Qiushi Engine drove the research loop from hypothesis formation and controlled ablations through training, evaluation, and reproducible release preparation.
Result
The complete local official-compatible evaluation is:
| Metric | Score |
|---|---|
| BLiMP | 68.5100 |
| BLiMP Supplement | 63.6400 |
| EWoK | 50.0200 |
| Entity Tracking | 28.3200 |
| COMPS | 52.0500 |
| GlobalPIQA | 38.5650 |
| (Super)GLUE | 69.8192 |
| Reading | 8.1650 |
| AoA | 0.0000 |
| Overall | 42.1210 |
The Fast checkpoint package contains all 133 expected prediction records, and AoA was measured over 19 checkpoints with 8,005 contexts per checkpoint. This is a local official-compatible result; leaderboard identity is established only after separate server-side submission and scoring.
Evaluation and checkpoints
- The complete full-plus-Fast prediction bundle is published under
submission/. - The 19 official Fast checkpoints are published as Hugging Face revisions:
chck_1Mthroughchck_10M, followed bychck_20Mthroughchck_100Min 10M increments. - See
CHECKPOINTS.md,TRAINING.md, andDATA.mdfor the public provenance and reproduction description.
Research contribution
The model combines two evidence-backed ideas found through Qiushi Engine's research:
- Compact-view reinvestment. Meaning-preserving compact views reduce redundant wording, and the recovered fixed word budget is reinvested in diverse natural-language experience. The full training pool remains within 10M words.
- Dual-path residual consolidation. An existing residual pathway preserves broad competence while a zero-output residual pathway is attached and trained during a short coherent continuation. Its calibrated contribution balances adaptation against retention without replacing the original pathway.
The tokenizer has 16,384 entries and was trained only on the same counted 10M-word training pool. The released endpoint represents 86,005,295 counted word exposures. Compact views were generated from training-corpus sentences with Qwen3.5-9B and were counted inside the data budget.
Architecture
- DeBERTa-v2 encoder: 8 layers, hidden size 480, 8 attention heads
- Two 128-dimensional bottleneck residual adapters per encoder layer
- 36,458,592 parameters
- Masked language modeling with whole-word masking
- Maximum training sequence length: 256 tokens
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo = "leslie721007/Qiushi-BabyLM-36M-Strict-Small-v4"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)
model.eval()
trust_remote_code=True is required because the residual architecture extends
the native Transformers DeBERTa-v2 implementation.
Scope and limitations
This is an English masked language model for representation learning, scoring, and fill-mask experiments. It is not an instruction-tuned conversational model. The release documents one high-performing endpoint; broader claims about the general mechanism require additional seeds and model families.