GPT-2 (124M), trained from scratch: chat model
zuu007/gpt2-from-scratch after supervised fine-tuning on databricks/databricks-dolly-15k (loss on the answers only). Part of github.com/zuurashad/gpt2-from-scratch, a PyTorch reproduction of
GPT-2 small trained from scratch on a single 6GB laptop GPU.
Try it in your browser.
The architecture is exactly GPT-2 small: 12 layers, 12 heads, width 768, a 1024-token
context and the GPT-2 BPE tokenizer. The weights load into Hugging Face's
GPT2LMHeadModel unchanged.
Files
| file | what it is |
|---|---|
model.safetensors | full-precision (fp32) weights |
onnx/model_quantized.onnx | per-channel int8 ONNX for the browser demo. Validation loss vs fp32: +0.0013 generating token by token (8,192 FineWeb-Edu tokens), +0.0058 over one forward pass (51,200 tokens) |
training.json | training step, validation loss and the training arguments |
Use it
from transformers import pipeline
generate = pipeline("text-generation", model="zuu007/gpt2-from-scratch-chat")
print(generate("<|system|>\nYou are a helpful assistant.<|endoftext|><|user|>\nWhat is photosynthesis?<|endoftext|><|assistant|>\n", max_new_tokens=60)[0]["generated_text"])
The model expects the chat template it was fine-tuned on (as above): each turn is <|system|>, <|user|> or <|assistant|> plus a newline and the text, ended by the single token <|endoftext|> (id 50256), where it also stops. The role markers are plain text, not special tokens.
In the browser, with transformers.js:
await AutoModelForCausalLM.from_pretrained("zuu007/gpt2-from-scratch-chat", { dtype: "q8" }).
Limitations
This is a 124M-parameter model trained on 2.5B tokens. It writes fluent English, but it is frequently wrong, especially about facts, arithmetic and recent events. Its only alignment is supervised fine-tuning on a small instruction dataset, so it can produce incorrect or inappropriate text.
Known limitation: follow-up questions. The chat model was fine-tuned only on single-turn examples (Dolly-15k), so it never learned to prioritise the newest message in a conversation. With 124M parameters it tends to continue the topic that fills most of its context, so an unrelated follow-up often gets an answer about the earlier question. Press Clear before changing topic. Multi-turn fine-tuning data (e.g. OpenAssistant) would be the fix.
Data and licences
- Pretraining: FineWeb-Edu (ODC-By 1.0)
- Fine-tuning (chat model only): databricks-dolly-15k, licensed CC BY-SA 3.0. Treat the chat weights as share-alike.
- Code: MIT, derived in part from Andrej Karpathy's build-nanogpt (MIT)