zondaxyz/babyconceptLM-baike102M_bpe60
This repository contains babyconceptLM-baike102M_bpe60, a Chinese Concept model trained on Baike102M BPE data.
The model uses custom Transformers code. Load it with trust_remote_code=True.
from transformers import AutoModel, AutoTokenizer
repo_id = "zondaxyz/babyconceptLM-baike102M_bpe60"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
Model Details
- Architecture:
ConceptDominantGPTBertForPreTraining - Hidden size: 768
- Vocabulary size: 16,384
- Max position embeddings: 512
- Pretraining sequence length: 256
- Token layers: 2
- Concept layers: 4
- Readout layers: 2
- Attention heads: 12 token heads, 14 concept heads, 12 readout heads
- Concept size: 896
- Intermediate sizes: 3,072 token/readout, 3,584 concept
- Model variant:
m18_2tok_4concept_2readout_mup_concept_main_gate_baike102m_bpe60 - Main gate:
concept_main_shell_residual - Segment target: 4.0 tokens per concept
- Training data recorded in local metadata:
train_zho_baike102m_m15_bpe_tokenized.bin - Tokenizer recorded in local metadata:
tokenizer_zho_baike102m_m15_bpe.json - Planned training steps: 64,277
- Training epochs: 60
Evaluation
The following numbers were collected from local Chinese BabyLM evaluation artifacts on 2026-06-11. For MRI tasks, the reported best result is selected by the highest report-level mean score across chinese-babylm-eval-pipeline/results and chinese-babylm-eval-pipeline/results_mri_layer_sweep.
For CogBench fMRI tasks, the local cogbench-fmri-0415 data contains train and dev splits but no released test split. The word_fMRI and fMRI numbers below are therefore dev-set scores, not held-out test scores.
| Task | Best score | Metric | Selection |
|---|---|---|---|
| zhoBLiMP | 72.29 | average accuracy | temperature 1.00 |
| Hanzi structure | 57.75 | average accuracy | temperature 1.00 |
| Hanzi pinyin | 50.05 | average accuracy | temperature 1.00 |
| word_fMRI | 0.563524 | mean Pearson r over 6 subjects | best layers: 03/04/05/06/08 tied; max subject score: 0.6168 on S31_score.mat |
| fMRI | 0.100538 | mean Pearson r over 36 domain/subject files | best layer: 09; max file score: 0.2269 on Language/03_average.mat |
| OCNLI | 63.19 | accuracy | fine-tuned dev score |
| TNEWS | 53.73 | accuracy | fine-tuned dev score |
| CLUEWSC2020 | 63.49 | accuracy | fine-tuned dev score |
| AFQMC | 68.98 | accuracy | fine-tuned dev score |
zhoBLiMP Field Accuracy
| Field | Accuracy |
|---|---|
| ellipsis | 67.00 |
| control_raising | 79.75 |
| BA | 78.41 |
| question | 82.54 |
| npi_licensing | 48.26 |
| nominal_expression | 82.48 |
| relativization | 86.25 |
| verb_phrase | 85.14 |
| anaphor | 34.83 |
| fci_licensing | 96.87 |
| topicalization | 96.08 |
| passive | 48.97 |
| quantifiers | 41.33 |
| argument_structure | 62.38 |
| classifier | 64.56 |
MRI Layer Sweep
| Layer | word_fMRI mean | fMRI mean |
|---|---|---|
| 00 | 0.558891 | 0.085080 |
| 01 | 0.560022 | 0.087955 |
| 02 | 0.561925 | 0.093293 |
| 03 | 0.563524 | 0.091392 |
| 04 | 0.563524 | 0.091048 |
| 05 | 0.563524 | 0.089261 |
| 06 | 0.563524 | 0.090662 |
| 07 | 0.563463 | 0.100302 |
| 08 | 0.563524 | 0.100186 |
| 09 | 0.560945 | 0.100538 |
Intended Use
This model is intended for Chinese BabyLM challenge. It is not intended as a production conversational assistant.