GPT-S-5M

GPT-S-5M is a first-generation model in the GPT-S small-model family: 5M parameters, 25B training tokens, a custom 4K tokenizer, 9 layers, and our all new T-X3 architecture, trained from scratch on a 3-source corpus.

At 5M parameters, GPT-S-5M achieves best-in-class performance among the small open models evaluated in this benchmark set, outperforming models 6x larger.

Benchmarks

All models evaluated in bf16 with an internal harness modeled on EleutherAI/lm-eval-harness. Scores are zero-shot; normalized accuracy is used where available.

Architecture

Config

vocab_size       = 4,096
hidden_size      = 192
num_layers       = 9
num_heads        = 6
num_kv_heads     = 2
head_dim         = 32
intermediate     = 672
block_size       = 512
rope_theta       = 2,500

Training

GPT-S-5M was trained for 25B tokens with a mixture built around educational web text, synthetic textbook-style material, and higher-quality web text.

Hyperparameters

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "AxiomicLabs/GPT-S-5M"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        repetition_penalty=1.1,
        no_repeat_ngram_size=4,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

This is a small base language model. It is not instruction tuned, has limited factual capacity, and uses a 512 token context window.