GPT-2 (124M) Trained From Scratch on TinyStories

This is an autoregressive GPT-2 model (124 million parameters) built and pretrained completely from scratch in PyTorch.

Model Details

• Architecture: GPT-2 (12 Layers, 12 Attention Heads, 768 Embedding Dimension)
• Vocabulary: 50,257 (GPT-2 BPE)
• Context Length: 256 tokens
• Training Dataset: TinyStories (~4.7M tokens)
• Hardware: Apple Silicon (MPS)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "sampatil1010/gpt2-124m-tinystories"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "Once upon a time, there was a little girl named Lily"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=60,
        temperature=0.7,
        top_k=40,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))