Block Attention Residuals · 10M · 32 layers · seed 1337
This model accompanies What Does Input-Dependent Depth Routing Buy in Data-Limited Language Modeling? for BabyLM 2026. It is a causal language model trained on the official BabyLM 10M-word training split, intended for research on language modeling with limited data and depth routing.
Each target sublayer computes input-dependent softmax weights over depth-wise source representations. Completed residual blocks provide additional sources.
Code and reproduction instructions
Architecture
| Setting | Value |
|---|---|
| Architecture | Block Attention Residuals |
| Transformer layers | 32 |
| Hidden size | 512 |
| Attention heads | 8 |
| Parameters | 116,295,168 |
| Vocabulary | 16,000 byte-level BPE tokens |
| Context length | 512 tokens |
| Checkpoint precision | float32 |
| Residual block size | 8 sublayers |
The shared backbone uses RMSNorm, SwiGLU, rotary positional embeddings, gated attention, and tied input/output embeddings.
Training
| Setting | Value |
|---|---|
| Training track | BabyLM 10M words |
| Optimizer updates | 471 |
| Seed | 1337 |
| Tokens per optimizer update | 262,144 |
| Optimizer | AdamW |
| Training dropout | 0.1 |
| Total training tokens processed | 123,469,824 |
| Total training words processed | 90,343,884 |
The training track denotes the size of the source corpus; training revisits that corpus. Dev and test data use the official independent splits. The tokenizer used during training is included in this repository.
Load with Transformers
Tested with Transformers 4.51.3 and PyTorch 2.7.x. The repository includes the model implementation, loaded with trust_remote_code=True.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "ypwhere/attnres4babylm2026-10m-l32-dynamic-s1337"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, trust_remote_code=True
).eval()
inputs = tokenizer("The child reads a book", return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits
next_token = logits[0, -1].argmax().item()
print(tokenizer.decode([next_token]))
Keep inputs within 512 tokens. The implementation supports causal scoring and generation without an efficient KV cache. This is a pretrained language model rather than an instruction-tuned chat model.
Evaluation
The paper uses the BabyLM evaluator at commit 3d57ddc. Comparisons across evaluator versions may differ because of changes to datasets or scoring. The accompanying model collection includes the paired architectures and seeds used for those comparisons.
Verify downloaded files
SHA256SUMS lists one SHA256 digest and its exact filename per line. The weights are stored in model.safetensors; the entry for that filename checks the downloaded weights. Other entries check the configuration, tokenizer, model code, and license files individually. From the downloaded model directory, run:
sha256sum -c SHA256SUMS # Linux
# macOS: shasum -a 256 -c SHA256SUMS
An OK result for each file confirms that it matches this model package. Download all files from the same repository revision when checking the hashes.
License
Model weights and accompanying code are released under the MIT license. The model implementation builds on nanoGPT.