Chinese BabyLM NanoChat d22 Step 10000

Model Summary

chinese-babylm-nanochat-d22-step10000 is a Chinese base causal language model trained with the karpathy/nanochat codebase for the NLPCC 2026 Chinese BabyLM shared task.

This is a base continuation model, not an instruction-tuned or chat-aligned assistant. It should be prompted with plain text and interpreted as next-token continuation. It has not gone through SFT, RLHF, DPO, or safety alignment.

Competition Compliance

This checkpoint is intended for the Chinese BabyLM Challenge final model submission.

Final submission repositories for the challenge should be public before the organizer deadline.

Model Details

Architecture

The model configuration:

This repository includes custom model and tokenizer code:

configuration_nanochat.py
modeling_nanochat.py
tokenization_nanochat.py

Loaders must use trust_remote_code=True.

Repository Files

Expected files in this model repository:

config.json
pytorch_model.bin
tokenizer.pkl
tokenizer_config.json
special_tokens_map.json
configuration_nanochat.py
modeling_nanochat.py
tokenization_nanochat.py
README.md
.gitattributes
img/baseline_task_comparison.png

pytorch_model.bin is the model weight file. tokenizer.pkl is required by the custom tokenizer.

Training Data

The model was trained on the official Chinese BabyLM corpus:

chinese-babylm-org/babylm-zho-100M

The corpus was converted into nanochat parquet shards:

Total prepared shards: 17.

No extra instruction data was used for this checkpoint.

Tokenizer

A 32K Chinese tokenizer was trained from the Chinese BabyLM training data.

Tokenizer compression on Chinese BabyLM text:

This tokenizer is specialized for Chinese BabyLM data. It gives much better compression on the target corpus than GPT-2 or GPT-4 tokenizers, at the cost of weaker general-purpose compression on English, code, math, and mixed-domain text.

Training Setup

The final recovered training run used one RTX 5090 GPU.

Main training configuration:

Official Open Evaluation

The model was exported to HuggingFace format and evaluated with a fresh copy of the official Chinese BabyLM eval pipeline:

SiyuanSong2004/chinese-babylm-eval-pipeline

The final official run used the unmodified official pipeline with backend: causal. The Hanzi tasks use the refreshed June 6 evaluation data/rules released by the Chinese BabyLM organizers. For cognitive evaluation, this repository exposes layer 8 as the HF AutoModel representation (representation_layer=8, representation_hidden_states_mode=selected_only). The language-model weights are still the same d22 step10000 checkpoint.

The repository root includes config.yaml, which freezes the official evaluation pipeline settings and NLU fine-tuning hyperparameters for hidden evaluation. It is an evaluation config, not a model architecture or training config.

Final gathered official open-task scores:

Baseline Comparison

The figure compares this model against three representative baselines:

• bert-base-chinese: strong pretrained encoder baseline.
• Qwen3-0.6B: strong generative baseline.
• babylm-chinese-bert-14m-epoch20: strong small BabyLM-trained baseline.

Our model is shown with the darkest pink. Its Hanzi scores use the June 6 refreshed evaluation data; baseline values are the public reference scores listed in the challenge materials available during development.

Usage

Because this repository contains custom model/tokenizer code, use trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "l0ulan/chinese-babylm-nanochat-d22-step10000"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "春天来了，公园里的花"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.8,
        top_k=50,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=False))

Limitations

• This is a base model, not a chat model. It does not reliably follow instructions.
• Outputs may be unrelated to user intent if prompted like an assistant.
• The model has not been safety-aligned.
• The training corpus is Chinese BabyLM-focused; general-domain multilingual performance is not the goal of this checkpoint.
• The model requires custom code loading with trust_remote_code=True.

Recommended Use

Appropriate uses:

• Chinese BabyLM research.
• Base LM continuation experiments.
• Tokenizer/model architecture experiments.
• Downstream SFT or probing baselines.
• Re-running the official Chinese BabyLM eval pipeline.

Not recommended as-is:

• Production assistant use.
• Safety-critical applications.
• Instruction-following/chat evaluation before SFT.

Acknowledgements

This checkpoint was trained with the karpathy/nanochat codebase and evaluated with the Chinese BabyLM eval pipeline from SiyuanSong2004/chinese-babylm-eval-pipeline.