Chinese BabyLM Cog A197 Strict Best
Model Summary
chinese-babylm-cog-a197-strict-best is a custom BERT-based wrapper model for the NLPCC 2026 Chinese BabyLM shared task, optimized for the Cognitive Modeling (Cog) Track. It combines internal strict BERT branches, official-corpus static features, and MLM uncertainty features for brain-aligned fMRI evaluation.
Competition Compliance
| Requirement | Status |
|---|---|
| From scratch | Source weights from internal random-initialized strict lineage only |
| No pretrained checkpoint | No official baseline or external pretrained checkpoint |
| No distillation | No teacher-model distillation |
| Data option | Official corpus (chinese-babylm-org/babylm-zho-100M) |
| Evaluation leakage | No CogBench stimulus text, labels, predictions, or item-level feedback used in training |
| No external resources | No pypinyin, IDS/CJKVI, radical tables, glyph/font resources, or inference-time rules |
| Frozen submission | Single exported wrapper checkpoint |
Architecture
| Field | Value |
|---|---|
| Model type | CogAllLayerUncertaintyGate wrapper |
| Architecture | CogAllLayerUncertaintyGateModel |
| Backend | mlm |
| Hidden size | 1409 |
| Max positions | 256 |
| Vocabulary | 16000 (WordPiece, trained on official corpus) |
Components
Word branch: A180 strict word BERT, layers [2,3,4] concat, RI scale 3.05 + log-frequency features.
Sentence branch: A130 + gamma -5.1 × (A158_s0250 − A122_s1500), layers [6,7,8] weighted [0, 0.98, 0.02] + raw384 window8 + log-frequency features.
Uncertainty: A129 step_01000 MLM features (logprob, surprisal, entropy, margin, probability), scale 1.25.
Hidden Evaluation
Evaluated with the final Chinese BabyLM pipeline (chinese-babylm/chinese-babylm-pipeline-final):
| Task | Score |
|---|---|
| word_fmri | 0.5655 |
| fmri | 0.1138 |
Usage
from transformers import AutoModel, AutoTokenizer
repo_id = "LikC1606/chinese-babylm-cog-a197-strict-best"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True, output_hidden_states=True)
inputs = tokenizer("春天的花开了。", return_tensors="pt")
outputs = model(**inputs, output_hidden_states=True)
Limitations
- Feature extraction model only, not for text generation.
- Requires
trust_remote_code=True. - Optimized for Cog track; not intended as a general NLU model.