GPT124M from scratch: 1B-token educational demo
This 124,439,808-parameter GPT-2-style decoder was initialized randomly and trained on 1,000,341,504 next-token predictions. No pretrained model weights were used. Only the GPT-2 tokenizer was reused. It completes English text; it is not a chat or instruction-following assistant.
Architecture and training
12 layers, 12 attention heads, hidden size 768, vocabulary 50,257, context length 1,024; learned positional embeddings and tied input/output embeddings. Dropout was zero. Training used two Tesla T4 GPUs on free Kaggle resources, FP16 autocast with FP32 parameters, AdamW (betas 0.9/0.95, epsilon 1e-8), weight decay 0.1 on matrix weights and gradient clipping at 1. Global batch: 524,288 tokens. Peak learning rate: 0.0006; linear warmup over 350M tokens followed by cosine decay on a fixed 10B-token horizon. Training stopped at the 1B milestone; it did not train for 10B tokens.
Full-state checkpoints and exact fresh-session numerical replay were verified during training. This repository contains inference weights only in FP32 safetensors, configuration and tokenizer assets. It does not contain optimizer state, RNG state, data shards or private notebook references.
Data and attribution
Contains information learned from FineWeb-Edu, Lozhkov et al., Hugging Face, 2024, DOI 10.57967/hf/2497. English sample-10BT, pinned source revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9; a bounded stream supplied 1.1B training token positions. No personal files were included. Documents with SHA256(text)'s first eight bytes, interpreted little-endian, modulo 100 equal to zero were assigned to held-out data; other documents were assigned to training. Identical documents therefore stayed in the same split. No additional near-duplicate decontamination was performed beyond upstream processing.
The source database is made available under ODC-By 1.0, with Common Crawl terms also relevant. Underlying web content retains its own rights. That database license is not asserted as a blanket license for these model weights or underlying content. No explicit license grant for the weights has been assigned in this release; contact the repository owner for reuse licensing. GPT-2 tokenizer assets retain OpenAI's MIT notice in TOKENIZER_LICENSE.txt.
Evaluation and limitations
Final held-out cross-entropy: 3.595338761806488; perplexity: 36.42803832595232, over 65,536 in-domain held-out token predictions. Initial held-out loss was 10.990954279899597. Final training-batch loss was 3.6821436882019043. These are diagnostic measurements, not general capability benchmarks. Sustained continuation throughput was 15,683 tokens/second on the two T4s.
The model is small and trained briefly. Outputs can be repetitive, inaccurate, biased or inappropriate, and it may reproduce source-like passages. It has no instruction tuning, factual verification or moderation guarantees. It is intended for an educational text-completion demonstration.
Local inference
import torch
from transformers import AutoTokenizer, GPT2LMHeadModel
repo = "felixmurraytang/gpt124m-scratch-finewebedu-1b-20261006"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = GPT2LMHeadModel.from_pretrained(repo, dtype=torch.float32).eval()
inputs = tokenizer("The purpose of education is", return_tensors="pt")
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=64, do_sample=False,
pad_token_id=tokenizer.eos_token_id, use_cache=True)
print(tokenizer.decode(output[0], skip_special_tokens=True))
from_pretrained here loads this newly trained artifact, not original GPT-2 pretrained weights. Tested export environment: PyTorch 2.11.0 and Transformers 5.16.1. No remote custom code is required. Keep prompt plus output within 1,024 tokens.