Quotes GPT Transformer

A character-level GPT model trained on Positive Quotes dataset.

Model Description

This model is a character-level transformer based on the GPT architecture, trained on a collection of positive quotes. It uses multi-head self-attention with causal masking to generate text autoregressively.

Model Architecture

• Tokenization: Character-level
• Architecture: Decoder-only Transformer (GPT)
• Attention: Multi-head self-attention with causal masking
• Layers: 6 transformer blocks
• Attention Heads: 6 heads
• Embedding Dimension: 384
• Context Length (Block Size): 512
• Dropout: 0.2
• Vocabulary Size: 94

Training Details

• Total parameters: 10,909,534
• Optimizer: AdamW
• Learning Rate: 0.0003
• Batch Size: 128
• Training Steps: 11000
• Loss Function: Cross-entropy

Training Results

• Final train loss and val loss were computed after training.

Intended Use

This model is intended for educational purposes and text generation experimentation. It generates character-level text based on the patterns learned from the quotes dataset.

Limitations

• The model is character-level, which can lead to slower generation compared to token-level models.
• The model may generate repetitive or nonsensical text.
• The model was trained on a relatively small dataset of quotes.

How to Use

import torch
import json

# Load config
with open("config.json") as f:
    config = json.load(f)

# Load vocab
with open("vocab.json") as f:
    vocab = json.load(f)

stoi = vocab["stoi"]
itos = {int(k): v for k, v in vocab["itos"].items()}
decoder = lambda s: "".join([itos[c] for c in s])
encoder = lambda s: [stoi[c] for c in s]

# Rebuild model (use same class definitions from the training script)
model = gpt(config["vocab_size"])
model.load_state_dict(torch.load("model.pt"))
model.eval()

# Generate text
context = torch.zeros((1, 1), dtype=torch.long)
output = decoder(model.generate(context, max_new_token=500)[0].tolist())
print(output)

Files

• model.pt - Model state dictionary
• config.json - Model hyperparameters
• config.yml - Hugging Face YAML metadata
• vocab.json - Character to index and index to character mappings
• generated_output.txt - Sample generated text

Citation

If you use this model, please cite the dataset: Colby/quotes from Hugging Face Datasets.