Tiny Qwen3 30M
Tiny Qwen3 30M is a 29.57M-parameter English causal language model trained from scratch on the TinyStories dataset. It is intended as a compact educational model for studying tokenizer training, pretraining, checkpointing, and text generation with the Hugging Face ecosystem.
This is a base language model, not an instruction-tuned or chat model.
Dataset
The model uses roneneldan/TinyStories, a synthetic English dataset of short stories generated to contain vocabulary and concepts that are typically understood by young children. Both the upstream train and validation splits are read in streaming mode.
Data preparation:
- The training stream is shuffled with seed 42 and a 10,000-example shuffle buffer.
- Empty stories are discarded.
- Stories are tokenized without automatically added special tokens, then separated with
<|endoftext|>. - Tokens from consecutive stories are concatenated and packed into fixed 512-token sequences without padding.
- Validation uses the same packing procedure and is limited to 256 packed sequences (131,072 tokens) per evaluation.
Tokenizer
A byte-level BPE tokenizer is trained from scratch on 100,000 TinyStories training examples. It uses NFC normalization, an 8,192-token vocabulary, and the special tokens <|endoftext|>, <unk>, <|im_start|>, and <|im_end|>. The tokenizer supports sequences up to 1,024 tokens, while training uses 512-token sequences.
Model architecture
| Parameter | Value |
|---|---|
| Architecture | Qwen3ForCausalLM |
| Parameters | 29,567,616 |
| Vocabulary size | 8,192 |
| Hidden size | 512 |
| Transformer layers | 9 |
| Attention heads | 8 |
| Key/value heads | 2 (GQA) |
| Head dimension | 64 |
| MLP intermediate size | 1,408 |
| Maximum position embeddings | 1,024 |
| RoPE theta | 1,000,000 |
| Tied input/output embeddings | Yes |
Training method
The model is pretrained from random initialization with the standard next-token causal language-modeling objective. Training uses the Hugging Face Trainer, streaming packed data, mixed precision when supported, gradient accumulation, periodic validation, and checkpoint-based resume.
Training parameters
| Parameter | Value |
|---|---|
| Optimizer | AdamW (adamw_torch_fused on CUDA) |
| Optimizer steps | 20,000 |
| Sequence length | 512 |
| Micro-batch size | 8 sequences/device |
| Gradient accumulation | 8 steps |
| Effective batch size | 64 sequences / 32,768 tokens |
| Learning rate | 5e-4 |
| Scheduler | Cosine |
| Warmup steps | 200 |
| Weight decay | 0.1 |
| Adam betas | (0.9, 0.95) |
| Gradient clipping | 1.0 |
| Precision | BF16 when supported; otherwise FP16 on CUDA or FP32 on CPU |
| Evaluation interval | 250 steps |
| Checkpoint interval | 250 steps |
| Random seed | 42 |
The recorded run completed 20,000 optimizer steps on an NVIDIA GeForce RTX 4070, with a final aggregate training loss of approximately 0.1317. This value is reported for reproducibility and should not be treated as a benchmark against other models.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "richarddzh/tiny-qwen3-30m"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
prompt = "Once upon a time,"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=80,
do_sample=True,
temperature=0.8,
top_p=0.9,
repetition_penalty=1.1,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Reproducibility
The complete training and upload workflow is available in the GitHub notebook.
Limitations
This small experimental model is trained only on synthetic English children's stories. It has limited factual knowledge, reasoning ability, context handling, and multilingual capability. Generated text may be repetitive, inconsistent, factually incorrect, or reflect artifacts in the source dataset. Do not use it for high-stakes decisions or assume its outputs are accurate.