BabyDeBERTa-24M (hanzi-aug v8s) — ChineseBabyLM 2026
Small from-scratch DeBERTa-v2 masked language model (~26M params) for the
ChineseBabyLM 2026 all-track challenge. Trained only on the official
chinese-babylm-org/babylm-zho-100M corpus (≤102M tokens) plus synthesized
HANZI auxiliary data (pinyin homophone + character-structure facts),
up-weighted during training via weighted sampling.
backend: mlm (this is a masked LM, evaluate with the MLM backend, not causal.)
Scores (full eval pipeline, step-030000)
| HANZI pinyin | HANZI structure | ZhoBLiMP | afqmc | ocnli | tnews | cluewsc2020 | word_fmri | fmri |
|---|---|---|---|---|---|---|---|---|
| 98.30 | 83.70 | 75.67 | 69.14 | 67.93 | 54.08 | 64.14 | 55.86 | 8.57 |
Aggregates: HANZI 91.0 · NLU 66.2 · Cog 32.2 → Overall ≈ 63.1
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ireallydk/babydeberta-24m-hanzi-v8s")
model = AutoModelForMaskedLM.from_pretrained("ireallydk/babydeberta-24m-hanzi-v8s")
Method
12k WordPiece tokenizer (NFKC, force-covers structure components, built only from
training data + general hanzi dictionaries — not the eval set). HANZI aux synthesized
from general sources (char↔pinyin homophones; whole↔component structure from
makemeahanzi + cjkvi-ids). Weighted sampling (aux_fraction=0.4). Anti-leakage:
corpus selection never uses the eval set; exact eval test sentences removed.
Training data + full pipeline: https://github.com/yjchen-713705/chinese-babylm-2026
(branch babydeberta-24m-hanzi-v8s).