Qiushi-BabyLM-34M Strict-Small v1
This is a 34.47M-parameter English masked language model developed by Qiushi Engine for the BabyLM 2026 Strict-Small track.
The model is a DeBERTa-v2 encoder trained from random initialization. Qiushi Engine carried out the corpus construction, tokenizer training, pretraining, evaluation, and release audit.
Model
| Property | Value |
|---|---|
| Architecture | DeBERTa-v2 masked language model |
| Parameters | 34,467,424 |
| Layers | 8 |
| Hidden size | 480 |
| Attention heads | 8 |
| FFN size | 1,920 |
| Vocabulary | 16,384 byte-level BPE tokens |
| Training sequence length | 256 |
| Objective | 15% whole-word masking |
Training
The tokenizer was trained only on the same 10,000,000-word pool used for model pretraining. The model then received exactly 100,000,000 word exposures. The corpus combines permitted English sources with compact text-only semantic views generated by Qwen3.5-9B. No teacher weights, hidden states, logits, or tokenizer were transferred to the student.
The fixed-budget corpus design compresses redundant semantic views and reinvests the saved word budget in additional source diversity. Training used AdamW, a peak learning rate of 0.001, weight decay 0.01, 6% warmup, cosine decay, batch size 256, and fixed 15% whole-word masking. The primary initialization seed was 43022.
Evaluation
The following values come from a complete local run of the official-compatible BabyLM 2026 Strict-Small evaluation and collation pipeline. They are not represented as official leaderboard scores until the leaderboard server accepts and scores the submitted predictions.
| Metric | Score |
|---|---|
| BLiMP | 65.8707 |
| BLiMP Supplement | 61.1657 |
| EWoK | 50.3932 |
| Entity Tracking | 27.4008 |
| COMPS | 52.0083 |
| (Super)GLUE | 70.2799 |
| GlobalPIQA | 36.0631 |
| Reading | 8.1382 |
| AoA | 0.0000 |
| Overall | 41.2578 |
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "leslie721007/Qiushi-BabyLM-34M-Strict-Small-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForMaskedLM.from_pretrained(repo_id)
text = f"The child {tokenizer.mask_token} the book."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
This is a research masked language model, not an instruction-tuned conversational assistant. It should not be used as a source of factual, medical, legal, or safety-critical advice.
License
This model is released for public research evaluation. Rights and conditions attached to third-party source corpora remain unaffected.
Reproducibility
SHA256SUMS records the exact hashes of the released model and tokenizer files. The public release intentionally excludes credentials, private system code, internal research traces, and operational logs.