gpt2-30m-fineweb-edu
A ~30M-parameter GPT-2 style decoder pretrained from scratch (random init) on
HuggingFaceFW/fineweb-edu
(sample-10BT config, streaming) in a single ~38-minute run on one NVIDIA A10G
as a budget demo of the full pretraining pipeline.
Architecture
- 6 layers, hidden 384, 6 heads, vocab 50257 (GPT-2 BPE tokenizer), context 512
- tied input/output embeddings, ~30.1M parameters
Training
- CLM loss on packed 512-token blocks, bf16
- AdamW β=(0.9, 0.95), weight decay 0.1, grad clip 1.0
- lr 6e-4, cosine decay, 150-step warmup, effective batch 64×512 = 32,768 tokens/step
- 4,000 optimizer steps ≈ 131M tokens seen (1 epoch over the streamed subset)
Results
| Metric | Value |
|---|---|
| final eval loss | 4.7510 |
| eval perplexity | ≈ 115.7 (exp of eval loss) |
| train loss (end) | ~4.9–5.0 |
The model is heavily undertrained (131M tokens for 30M params) — expect repetitive, low-coherence text. It demonstrates the pipeline, not quality.
Usage
from transformers import pipeline
pipe = pipeline("text-generation", "zhouaz/gpt2-30m-fineweb-edu")
print(pipe("The history of the Roman Empire begins", max_new_tokens=80, do_sample=True)[0])
Reproduction
Training script pattern follows transformers examples/pytorch/language-modeling/run_clm.py:
streaming dataset → online tokenization → block packing → Trainer with max_steps,
periodic push_to_hub (every 500 steps).