GPT-U-20M

A 20.5M-parameter Llama-architecture language model pretrained from scratch on 2.6B tokens of web text, educational text and code.

Architecture

The 16,384-token vocabulary keeps the embedding at 31% of the parameters (a 50k vocabulary would need 19.3M parameters at width 384) and lets tokens be stored as uint16.

Tokenizer. Byte-level BPE with a cl100k-style split that keeps whitespace runs whole (Python indentation), one token per digit, and <|endoftext|> (id 0) as BOS/EOS/padding. Round-trip decode(encode(x)) == x holds on 1,000 held-out samples (tabs, CRLF, emoji, CJK, RTL, control characters).

Training data

One epoch over 2.6B tokens, no repetition:

• Each source is streamed from the Hub and tokenized up to its token budget; the documents are then interleaved per document in a seeded random order, i.e. with fixed proportions for the whole run (no domain curriculum), so the realized shares match the targets exactly. (Holding the four streams open at once was avoided: it needed 7–8 GB of RAM and destabilized the training machine.)
• The Stack v3 is grouped by repository; repositories were split into individual files, vendored files dropped, and Python held at 26.6% of the code tokens (it is ~4% of Stack v3 naturally).
• Documents are separated by <|endoftext|> and packed into contiguous 1,025-token blocks (1,024 inputs + 1 shifted target), no padding. Batches are sampled from a seeded permutation of all blocks.
• Validation holdouts (2M tokens per domain) come from source files that never fed training or the tokenizer.

Training

Trained on 2× RTX 3060 12GB (no NVLink) under WSL2 with PyTorch DDP and torch.compile (mode: compile): 4.6 h of active training time, 167,893 tokens/s on average (MFU ≈ 52% of the GPUs' measured bf16 matmul throughput), max GPU temperature 76°C, 3 automatic resume(s) from checkpoints after the machine was shut down, 0 loss-spike rollback(s).

Results

Validation loss on the 2M-token holdouts (1,600 fixed 1,024-token windows per domain) and bits per byte, bpb = loss_nats × tokens / (bytes × ln 2), which is comparable across tokenizers:

The aggregate is the equal-weight mean of the three holdouts (same number of tokens each). Code tokens are far easier to predict than prose with this tokenizer, which pulls the aggregate down; the prose domains (3.26–3.50 nats/token) are the fairer comparison point for web-text models, and bits per byte is the fairer one across tokenizers.

Zero-shot benchmarks (lm-evaluation-harness). <|endoftext|> is prepended as BOS. hellaswag and winogrande sit at chance, as expected at 20M parameters; arc_easy and piqa land clearly above it, which reflects surface lexical knowledge more than reasoning. They are reported for transparency, not as a quality signal.

Sample generations (temperature 0.8, top-p 0.95) are in out/samples.md.

Loss curve

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("DedeProGames/GPT-U-20M")
model = AutoModelForCausalLM.from_pretrained("DedeProGames/GPT-U-20M")

inputs = tokenizer("The history of the Roman Empire", return_tensors="pt")  # prepends <|endoftext|> as BOS
output = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.95)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Checkpoints

• main: final weights (step 19,840, end of the decay phase).
• stable-15872: weights at step 15,872, the end of the constant-learning-rate phase, meant for continued pretraining (e.g. extending to 4–5B tokens before decaying again). Load with revision="stable-15872".

Limitations

• It is tiny. 20M parameters is a research and teaching scale; the model has shallow world knowledge and weak reasoning.
• Heavily over-trained for its size by design: 127 tokens per parameter, about 6× the Chinchilla-optimal ratio (~20).
• Not for production use. It hallucinates freely and produces fluent but often false statements.
• Generated code is not functional. It imitates the surface form of code; do not run it.
• English only (web and educational text); no instruction tuning, no safety tuning. Web data can carry biases and offensive content.
• Context is limited to 1,024 tokens.

Data provenance and licenses

The model weights are released under Apache-2.0. Tokenized data is not redistributed.

Reproducibility

Everything used to build this model is in the repository: scripts/ (tokenizer training, data preparation, training, evaluation, publishing), configs/model.json, logs/train.jsonl (every 10 steps: loss, lr, grad norm, throughput, MFU, GPU temperatures, memory), logs/tokenizer_report.json and out/data_index.json (token counts per shard and domain, holdout bytes). Raw evaluation outputs: out/eval.json and out/lm_eval_results.json.