Qiushi-BabyLM-34M Strict-Small v2
Qiushi-BabyLM-34M-Strict-Small-v2 is a 34.47M-parameter English masked language model developed by Qiushi Engine for public research on data-efficient pretraining.
This version evaluates U256 Faithful Experience Utilization, a controlled training-example construction. Long tokenized rows are segmented into continuation chunks of at most 256 tokens instead of silently dropping row tails. This exposes the model to all text already charged to the fixed word budget; it does not add training words, epochs, external labels, or a new objective.
Model
| Property | Value |
|---|---|
| Architecture | DeBERTa-v2 masked language model |
| Parameters | 34,467,424 |
| Layers | 8 |
| Hidden size | 480 |
| Attention heads | 8 |
| FFN size | 1,920 |
| Vocabulary | 16,384 byte-level BPE tokens |
| Training sequence length | Up to 256 tokens |
| Objective | 15% whole-word masking |
The architecture, tokenizer algorithm, objective, and optimizer are intentionally standard. The experimental variable is how already-budgeted training experience is made visible to the model.
The public method contract and standalone chunking reference are included in U256_METHOD.md and u256_reference.py.
Training
The tokenizer was trained only on the same 10,000,000-word pool used for pretraining. The model was initialized from random weights and received exactly 100,000,000 charged word exposures over 10 epochs.
The fixed-budget corpus combines permitted English sources with compact text-only semantic views generated by Qwen3.5-9B. No teacher weights, hidden states, logits, or tokenizer were transferred to the student. U256 increased visible BPE-token coverage by approximately 2.59% relative to row-level truncation while preserving the same corpus, word accounting, example multiset, model, tokenizer, masking objective, optimizer, and random-seed family.
Training used AdamW, a peak learning rate of 0.001, weight decay 0.01, 6% linear warmup followed by cosine decay, effective batch size 256, and fixed 15% whole-word masking. The initialization seed was 43022; framework and training-stream seeds were 43 and 43023.
Evaluation
The following scores come from a complete local run of the BabyLM 2026 Strict-Small official-compatible evaluation and collation pipeline. The run includes the full checkpoint ladder required for AoA. These values are not represented as official leaderboard scores until the public leaderboard accepts and scores the submitted predictions.
| Metric | Score |
|---|---|
| BLiMP | 66.8172 |
| BLiMP Supplement | 60.8477 |
| EWoK | 50.3861 |
| Entity Tracking | 28.4331 |
| COMPS | 51.6455 |
| (Super)GLUE | 70.3747 |
| GlobalPIQA | 36.1359 |
| Reading | 7.3229 |
| AoA | 0.0000 |
| Overall | 41.3292 |
Against its strictly matched baseline, U256 improved Overall by 0.0715, with gains in BLiMP and Entity Tracking and a material decline in Reading. It should therefore be interpreted as a measured experience-utilization tradeoff, not as a state-of-the-art claim or evidence that longer visibility universally improves downstream performance.
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "leslie721007/Qiushi-BabyLM-34M-Strict-Small-v2"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForMaskedLM.from_pretrained(repo_id)
text = f"The child {tokenizer.mask_token} the book."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
This is a research masked language model, not an instruction-tuned conversational assistant. It should not be used as a source of factual, medical, legal, financial, or safety-critical advice.
Reproducibility and privacy
The public package includes exact file hashes. Credentials, private system code, local paths, internal research traces, and operational logs are intentionally excluded.
License
The weights are released for public research evaluation. Rights and conditions attached to third-party source corpora and generated text remain unaffected.