A3-large — Chinese BabyLM 2026 (Hanzi track)

A from-scratch character-level GPT (123.4M params) for the Chinese BabyLM 2026 shared task, trained on the official babylm-zho corpus within the 102M-jieba-token budget (counted with jieba 0.42.1).

Architecture

• 16 layers, hidden 768, 12 heads, FFN 3072, sequence length 512; tied embeddings.
• Character-level tokenizer (vocab 8891) with radical/component characters and IDC operators in-vocab.
• Deterministic per-character IDS-component and structured pinyin (声母/韵母/声调) features fused into the input embeddings, plus auxiliary component- and pinyin-prediction heads (decayed during training).
• Custom code ships in the repo (modeling_hanzi.py, configuration_hanzi.py); load with trust_remote_code=True.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("EjZhou/chinese-babylm-2026-hanzi-a3large", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("EjZhou/chinese-babylm-2026-hanzi-a3large", trust_remote_code=True)

Also exposes AutoModelForMaskedLM and AutoModel (hidden states) for the cogbench / finetune tracks.

Evaluation

Score with the official pipeline and the bundled config.yaml:

python pipeline.py eval --config config.yaml

Zero-shot results (causal scoring): hanzi_structure 75.9 · hanzi_pinyin 100.0 · zhoblimp 78.4.

Training data

Official babylm-zho natural text plus deterministically-generated character-knowledge sentences (component structure + homophone relations) over a broad character inventory; total training corpus kept under the 102M-jieba budget. Exact benchmark sentences were excluded from training.