GPT-S-1.4M

GPT-S-1.4M a first-generation model in the GPT-S small-model family built on the T-X3 architecture: 1.4M parameters, 6B training tokens, a custom 4K tokenizer, 5 layers, and all new Exclusive Grouped-query Attention (XGQA), trained from scratch on a 5-source corpus.

See how it compares to similar models here: Open SLM Leaderboard

Benchmarks

All evaluations use zero-shot multiple-choice scoring. Normalized accuracy is reported where available.

Architecture

Config

vocab_size       = 4,096
hidden_size      = 128
num_layers       = 5
num_heads        = 4
num_kv_heads     = 2
head_dim         = 32
intermediate     = 341
block_size       = 384
rope_theta       = 2,500

Training

GPT-S-1.4M was trained from scratch for 6B tokens on a mixed English corpus built around educational web text, synthetic textbook-style material, and higher-quality web text.

Hyperparameters

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "AxiomicLabs/GPT-S-1.4M"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        repetition_penalty=1.1,
        no_repeat_ngram_size=4,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

This is a very small base language model. It is not instruction tuned, has limited factual capacity, and uses a 384-token context window.