RWKV-7 Half-CLM — BabyLM 2026 Strict-Small
A ~28M-parameter RWKV-7 recurrent (subquadratic) language model trained from scratch on the BabyLM 2026 Strict-Small corpus (14.87M tokens, 32K BPE). Causal CLM, zero auxiliary objectives. Inference is a single forward (CLM) pass.
This is the primary competitive submission for the BabyLM 2026 Strict-Small track and the headline model from Halved CLM Exposure Mitigates Late-Training Collapse in Small Recurrent Language Models.
Method
Identical RWKV-7 architecture trained with halved CLM exposure: odd optimizer steps advance the learning-rate schedule without applying a gradient update ("step-skipping"). This reduced update pressure prevents the late-training collapse that the full-exposure baseline suffers, and does so without any auxiliary objectives (no MLM, MTP, or n-gram heads).
Performance (official BabyLM 2026 pipeline, strict-small)
Scores below are from a clean, end-to-end re-run of the official evaluation pipeline
(evaluation_pipeline.sentence_zero_shot / reading / finetune), zero manual editing of predictions.
All scores come from a single, self-consistent re-run (one set of predictions, no cherry-picking across runs):
| Metric | Score |
|---|---|
| BLiMP (filtered) | 68.70 |
| BLiMP Supplement | 57.24 |
| EWoK | 1.53 (form-degenerate; shared by all non-baseline teams) |
| Entity Tracking | 19.98 (near-chance; see note) |
| COMPS | 52.55 |
| Reading | 0.86 |
| GLUE | 58.72 |
| AoA | ~0 |
| Text Average (8-col) | 32.45 |
Note on Entity Tracking. Run through the official pipeline, these weights score 19.98 on the full ET set, and the number is deterministic: three independent re-runs give 19.98, and applying first-index versus random tiebreaks to the same captured log-probs gives 19.98 either way (100 random draws, std 0.00). Internal evaluation corroborates it (~18.6 at step 18k). An earlier standing submission recorded ET ~42, but that figure is not reproducible from these released weights and we do not carry it forward. We report 19.98, the model's actual output.
Architecture
- RWKV-7 (
RWKV7ForCausalLM), 8 layers, n_embd 384, vocab 32,000 - Chunked WKV-7 kernel (diagonal-only within-chunk decay, chunk_size 64) — disclosed approximation
- Pure-PyTorch modeling for HF eval compatibility (ported from the MLX training implementation)
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"yashmahe2018/rwkv7-halfclm-strict-small", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
"yashmahe2018/rwkv7-halfclm-strict-small", trust_remote_code=True
)
Citation
From the BabyLM 2026 submission Halved CLM Exposure Mitigates Late-Training Collapse in Small Recurrent Language Models.