BiRWKV-7 — BabyLM 2026 Strict-Small (step 14k, peak Entity Tracking)

Architecture: BiRWKV-7 (~28M parameters) — bidirectional RWKV-7
Checkpoint: Step 14,000 — peak Entity Tracking checkpoint
Fused inference ET: 40.20% (internal eval; official CLM-only ET: 22.45%)
BLiMP at step 14k: 68.83%

This checkpoint is notable for the divergence between CLM-only and fused inference Entity Tracking: fused inference (forward + reversed backward pass) reaches 40.20% ET at step 14k, competitive with BLaLM (40.93%), while CLM-only scoring gives 22.45% — demonstrating that the bidirectional representation encodes entity state that CLM scoring cannot access.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "yashmahe2018/birwkv7-babylm2026-ck14k",
    trust_remote_code=True
)

Citation

@misc{maheshwari2026halved,
  title={Halved CLM Exposure Mitigates Late-Training Collapse in Small Recurrent Language Models},
  author={Maheshwari, Yash},
  year={2026},
  note={BabyLM 2026 submission}
}