zondaxyz/babyconceptLM-baike102M_bpe50
This repository contains babyconceptLM-baike102M_bpe50, a Chinese Concept model trained on Baike102M BPE data.
The model uses custom Transformers code. Load it with trust_remote_code=True.
from transformers import AutoModel, AutoTokenizer
repo_id = "zondaxyz/babyconceptLM-baike102M_bpe50"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
Model Details
- Architecture:
ConceptDominantGPTBertForPreTraining - Hidden size: 768
- Vocabulary size: 16,384
- Max position embeddings: 512
- Pretraining sequence length: 256
- Token layers: 2
- Concept layers: 4
- Readout layers: 2
- Attention heads: 12 token heads, 14 concept heads, 12 readout heads
- Concept size: 896
- Intermediate sizes: 3,072 token/readout, 3,584 concept
- Main gate:
concept_main_shell_residual - Segment target: 4.0 tokens per concept
- Training data recorded in local metadata:
train_zho_baike102m_m15_bpe_tokenized.bin - Tokenizer recorded in local metadata:
tokenizer_zho_baike102m_m15_bpe.json - Planned training steps: 53,875
- Training epochs: 50
Evaluation
The following numbers were collected from local Chinese BabyLM evaluation artifacts on 2026-06-11. For MRI tasks, the reported best result is selected by the highest report-level mean score across chinese-babylm-eval-pipeline/results and chinese-babylm-eval-pipeline/results_mri_layer_sweep.
For CogBench fMRI tasks, the local cogbench-fmri-0415 data contains train and dev splits but no released test split. The word_fMRI and fMRI numbers below are therefore dev-set scores, not held-out test scores.
| Task | Best score | Metric | Selection | |
|---|---|---|---|---|
| zhoBLiMP | 71.27 | average accuracy | temperature 1.00 | |
| word_fMRI | 0.5624 | mean Pearson r over 6 subjects | best layer: 07; max subject score: 0.6114 on S31_score.mat | |
| fMRI | 0.0998 | mean Pearson r over 36 domain/subject files | best layer: 09; max file score: 0.2262 on Language/03_average.mat |
zhoBLiMP Field Accuracy
| Field | Accuracy |
|---|---|
| ellipsis | 68.33 |
| control_raising | 78.25 |
| BA | 77.49 |
| question | 82.33 |
| npi_licensing | 46.22 |
| nominal_expression | 81.85 |
| relativization | 83.25 |
| verb_phrase | 81.45 |
| anaphor | 35.17 |
| fci_licensing | 95.67 |
| topicalization | 96.08 |
| passive | 49.53 |
| quantifiers | 35.50 |
| argument_structure | 62.05 |
| classifier | 64.22 |
MRI Layer Sweep
| Layer | word_fMRI mean | fMRI mean |
|---|---|---|
| 00 | 0.5588 | 0.0850 |
| 01 | 0.5599 | 0.0878 |
| 02 | 0.5613 | 0.0938 |
| 03 | 0.5620 | 0.0914 |
| 04 | 0.5620 | 0.0920 |
| 05 | 0.5620 | 0.0900 |
| 06 | 0.5620 | 0.0928 |
| 07 | 0.5624 | 0.0997 |
| 08 | 0.5620 | 0.0995 |
| 09 | 0.5596 | 0.0998 |
Intended Use
This model is intended for Chinese BabyLM challenge. It is not intended as a production conversational assistant.