decoder-only-transformer-tinystories-32m
A custom GPT-style language model trained from scratch using PyTorch.
Model Details
| Parameter | Value |
|---|---|
| Architecture | GPT (Decoder-only Transformer) |
Hidden size (d_model) | 768 |
| Attention heads | 8 |
| Transformer blocks | 1 |
| Max sequence length | 1024 |
| Vocabulary size | 32000 |
| Dropout | 0.2 |
Tokenizer
Custom BPE tokenizer trained with the HuggingFace tokenizers library.
Special tokens: <|endoftext|> · <|pad|> · <|unk|>
Quick Start
You can easily load this model and tokenizer using the transformers library. Because the model uses a custom architecture, you must pass trust_remote_code=True.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("shubhamprakash108/decoder-only-transformer-tinystories-32m", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("shubhamprakash108/decoder-only-transformer-tinystories-32m", trust_remote_code=True)
# Set up device
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
# Generate text
prompt = "The transformer is based on"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
output_ids = model.generate(
**inputs,
max_new_tokens=50,
do_sample=True,
temperature=0.8,
pad_token_id=tokenizer.eos_token_id
)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
Training Details
- Optimizer: AdamW (lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)
- Scheduler: CosineAnnealingLR (eta_min=1e-5)
- Loss: CrossEntropyLoss (next-token prediction)
- Gradient clipping: max_norm=1.0
License
Apache 2.0