StoryBananaMind-7.5M

StoryBananaMind-7.5M is a small GPT-2 style causal language model trained for short English story generation on the TinyStories dataset.

Model Details

• Architecture: GPT2LMHeadModel
• Parameters: 7,428,960
• Vocabulary size: 8,192
• Context length: 256 tokens
• Hidden size: 288
• Layers: 5
• Attention heads: 6
• Weights format: safetensors
• Training dataset: roneneldan/TinyStories

Intended Use

This model is intended for lightweight English story generation experiments, educational use, and local text-generation testing. It is a very small model and should not be expected to perform broad instruction following, factual answering, or robust reasoning.

Usage

from transformers import AutoTokenizer, GPT2LMHeadModel
import torch

model_id = "StoryBananaMind-7.5M"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = GPT2LMHeadModel.from_pretrained(model_id).eval()

prompt = "Once upon a time, there was a little cat named Lily."
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.8,
        top_p=0.95,
        do_sample=True,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training

The model was trained from scratch with a custom GPT-2 tokenizer trained to an 8,192-token vocabulary. Training used fixed 256-token blocks from TinyStories.

Key configuration:

• n_layer: 5
• n_embd: 288
• n_head: 6
• n_inner: 1152
• n_positions: 256
• vocab_size: 8192

Files

• model.safetensors: model weights
• config.json: Transformers model configuration
• generation_config.json: default generation configuration
• tokenizer.json: tokenizer
• tokenizer_config.json: tokenizer metadata
• storybananamind-preview.png: model card preview image