Tiny Qwen3 30M Chinese
Tiny Qwen3 30M Chinese is a 29.57M-parameter Chinese causal language model trained from scratch on the Chinese Small LM Corpus. It is a compact educational model for experimenting with tokenizer training, language-model pretraining, checkpointing, and Chinese text generation in the Hugging Face ecosystem.
This is a base language model, not an instruction-tuned or chat model.
Dataset
The model uses the train split of richarddzh/chinese-small-lm-corpus. The dataset is shuffled with seed 42, after which 10,000 documents are reserved for validation and the remaining documents are used for training.
Data preparation:
- Empty texts are discarded.
- Texts are tokenized without automatically added special tokens and separated with
<|endoftext|>. - Tokens from consecutive documents are concatenated and packed into fixed 512-token sequences without padding.
- Validation uses the same packing procedure and is limited to 256 packed sequences (131,072 tokens) per evaluation.
Tokenizer
A byte-level BPE tokenizer is trained from scratch on up to 100,000 training documents, with each document truncated to 2,000 characters for tokenizer training. It uses NFC normalization, an 8,192-token vocabulary, a minimum token frequency of 2, and the special tokens <|endoftext|>, <unk>, <|im_start|>, and <|im_end|>. The tokenizer supports sequences up to 1,024 tokens, while pretraining uses 512-token sequences.
Model architecture
| Parameter | Value |
|---|---|
| Architecture | Qwen3ForCausalLM |
| Parameters | 29,567,616 |
| Vocabulary size | 8,192 |
| Hidden size | 512 |
| Transformer layers | 9 |
| Attention heads | 8 |
| Key/value heads | 2 (GQA) |
| Head dimension | 64 |
| MLP intermediate size | 1,408 |
| Maximum position embeddings | 1,024 |
| RoPE theta | 1,000,000 |
| Tied input/output embeddings | Yes |
Training method
The model is pretrained from random initialization with the standard next-token causal language-modeling objective. The notebook uses the Hugging Face Trainer, packed iterable datasets, mixed precision when supported, gradient accumulation, periodic validation, and checkpoint-based resume.
Training configuration
| Parameter | Value |
|---|---|
| Optimizer | AdamW (adamw_torch_fused on CUDA) |
| Maximum optimizer steps | 30,000 |
| Sequence length | 512 |
| Micro-batch size | 32 sequences/device |
| Gradient accumulation | 2 steps |
| Effective batch size | 64 sequences / 32,768 tokens |
| Learning rate | 5e-4 |
| Scheduler | Cosine |
| Warmup steps | 200 |
| Weight decay | 0.1 |
| Adam betas | (0.9, 0.95) |
| Gradient clipping | 1.0 |
| Precision | BF16 when supported; otherwise FP16 on CUDA or FP32 on CPU |
| Evaluation interval | 250 steps |
| Checkpoint interval | 250 steps |
| Random seed | 42 |
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "richarddzh/tiny-qwen3-30m-zh"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
prompt = "问题：太阳是地球的什么？回答：太阳是"
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=80,
do_sample=True,
temperature=0.8,
top_p=0.9,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Reproducibility
The complete data preparation, tokenizer training, model configuration, pretraining, evaluation, and upload workflow is available in the training notebook.
Limitations
This is a small experimental base model. Its factual knowledge, reasoning ability, instruction-following behavior, and context handling are limited. Generated text may be repetitive, inconsistent, factually incorrect, or reflect biases and artifacts in the training corpus. Do not use it for high-stakes decisions or assume its outputs are accurate.