Chinese BabyLM 2026 — DeBERTaV2 (v3)
Submission for the Chinese BabyLM 2026 Challenge (NLPCC 2026).
Model Description
A compact DeBERTaV2 masked language model pretrained from scratch on Chinese child-directed and child-produced text. Designed for data-constrained language modeling with a pure-Chinese WordPiece tokenizer.
- Architecture: DeBERTaV2ForMaskedLM (disentangled attention, relative position)
- Parameters: ~6M (256h / 4L / 4H / 1024ff)
- Tokenizer: WordPiece, vocab=8000 (limit_alphabet=2000), NFKC + lowercase
- Sequence length: 256 tokens
- Training: 5000 steps, batch=128, lr=5e-4, bf16, dynamic span masking (mean_span=3.0)
- Data: Full
chinese-babylm-org/babylm-zho-100Mcorpus (183K documents)
Results
Best model from a 2x3 ablation study (architecture x radical embedding):
| Model | NLU | HANZI | Cog | Overall |
|---|---|---|---|---|
| DeBERTaV2 (this model) | 60.22 | 84.15 | 31.14 | 58.50 |
| BERT plain | 58.49 | 85.25 | 31.41 | 58.38 |
| ModernBERT plain | 59.61 | 80.88 | 31.34 | 57.28 |
Key findings:
- Tokenizer granularity is the primary factor for HANZI track performance (8K vocab >> 32K vocab)
- DeBERTaV2 achieves the highest NLU scores (zhoblimp: 69.89)
- Radical embedding hurts performance with character-level tokenization
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained("exusiaiw/chinese-babylm-2026-v3")
tokenizer = AutoTokenizer.from_pretrained("exusiaiw/chinese-babylm-2026-v3")
Citation
Chinese BabyLM 2026 Challenge — NLPCC 2026 Workshop