gpt-bert-entmax-graded
BabyLM 2026 Strict-Small submission (NTUA). A GPT-BERT hybrid language model: gpt_bert_mixed — hidden 384, 12 layers, 6 heads, alpha-entmax attention with per-layer alpha schedule [1.25, 1.25, 1.25, 1.25, 1.75, 1.75, 1.75, 1.75, 1.5, 1.5, 1.5, 1.5].
- Training data:
BabyLM-community/BabyLM-2026-Strict-Small(≤100M words, counting repeated exposures, per CfP §4.2). Uniform ~10 passes via bin-packing (each segment once per epoch). - Objective: mixed masked+causal LM (causal_prob 0.0625, masked-heavy). Evaluate with the
mntpbackend. - Intermediate checkpoints: available as branches
chck_1M … chck_100M(words seen), per the BabyLM checkpoint convention.
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("JimK28/gpt-bert-entmax-graded", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained("JimK28/gpt-bert-entmax-graded", trust_remote_code=True)
Custom architecture — trust_remote_code=True is required (config auto_map points at
modeling_gpt_bert).
Requires pip install entmax (entmax attention). Trained with the NTUA BabyLM pipeline
(https://github.com/emhadzi/BabyLM).