deberta-base-75k-sam_ext-s2
Vision-seeded model with synthetically extended grounding. On top of the real region features, 1,151 previously unseeded concrete words were grounded by generating scene descriptions with an LLM, rendering them with a text-to-image model, localizing the target with an open-vocabulary detector, and pooling SAM ViT-B features inside the detected boxes (+737 seeded tokens).
Seed 2 of 3. This configuration was retrained with three random seeds: augustinian-babylm/deberta-base-75k-sam_ext-s1, augustinian-babylm/deberta-base-75k-sam_ext-s2, augustinian-babylm/deberta-base-75k-sam_ext-s3.
Baseline: augustinian-babylm/deberta-base-75k.
Part of the model set for Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model (paper, code).
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("augustinian-babylm/babylm-bpe-75k")
model = AutoModelForMaskedLM.from_pretrained("augustinian-babylm/deberta-base-75k-sam_ext-s2")
Training
DeBERTa-v3-base masked LM (768 hidden, 12 layers, 12 heads), trained for 10 epochs
on bb24.train, the ~9.9M-word corpus of Edman et al. (2024), which mixes
LLM-generated paraphrase data with portions of the official BabyLM corpus. This is
not the official 2026 strict-small distribution.
| Learning rate | 2e-4, cosine schedule, 4000 warmup steps |
| Optimizer | AdamW (beta1 0.9, beta2 0.95), weight decay 0.01 |
| Effective batch | 256 (gradient accumulation 4) |
| Context length | 64 to 128 from epoch 5, max 512 |
| Vocabulary | BPE, 75k merges (augustinian-babylm/babylm-bpe-75k) |
Only the input-embedding initialization differs between a vision-init model and its baseline. Every other hyperparameter, the data order, and the random seed are held fixed.
Checkpoints
Intermediate checkpoints are stored as branches, saved Pythia-style by step:
from transformers import AutoModelForMaskedLM
AutoModelForMaskedLM.from_pretrained("augustinian-babylm/deberta-base-75k-sam_ext-s2", revision="step256")
Call huggingface_hub.list_repo_refs("augustinian-babylm/deberta-base-75k-sam_ext-s2") to list the available steps.
Results
Vision seeding produces a consistent object-property gain (COMPS, +1.30 averaged over nine encoder-by-vocabulary configurations, positive in all nine) while grammar-focused benchmarks stay flat. On a corpus-tailored Visual-Property Swap probe the seeded model leads the baseline in all three training seeds; a pair-level re-analysis shows that the advantage cannot be attributed to the individual seeded words. Details: https://github.com/bylinina/augustinian_babylm
Limitations
Trained on a non-standard corpus (see above), so results are not directly comparable to submissions trained on the official 2026 strict-small data. Single masked-LM architecture. Research artifact, not intended for deployment.
Citation
@inproceedings{bylinina2026augustinian,
title = {Augustinian BabyLM: What Ostensive Definition Can and Cannot
Teach a Small Language Model},
author = {Bylinina, Lisa},
booktitle = {Proceedings of the BabyLM Workshop},
year = {2026},
url = {https://openreview.net/forum?id=B4TD4XdlwF}
}