GPT-S3-3M

GPT-S3-3M is Axiomic Labs' third-generation GPT-S base model, built on a hybrid GDN2/gated GQA architecture with Full Attention Residuals and trained from scratch with TrainWork. At 2.97M parameters, it achieves an Intelligence Index of 9.61 on the Open SLM Leaderboard achieving 1st in the <3m class and 2nd in <10m.

Performance

Relative Leaderboard Performance

The Intelligence Index chance-normalizes HellaSwag, combined ARC (the mean of ARC-Easy and ARC-Challenge), PIQA, and ArithMark-3, then applies weights of 1.00, 1.00, 1.00, and 0.65 respectively.

Architecture

Layer layout

embedding
  -> [GDN-2, GDN-2, GDN-2, gated GQA]   x 2
  -> RMSNorm -> tied LM head

Config

vocab_size        = 4,096     (digit-split byte-level BPE)
hidden_size       = 144
num_layers        = 8         (6 GDN-2 + 2 gated GQA)
gdn_heads         = 3 x 48
attention_heads   = 3 query / 1 KV, head_dim 48
rotary_dim        = 24
intermediate      = 352
block_size        = 1,024
rope_theta        = 10,000
total params      = 2,973,666

Parameter Breakdown

Training

GPT-S3 was trained for 114,441 steps on one GPU, with a global batch of 262,144 tokens per step and standard next-token prediction throughout. The complete run processed 30B tokens; the released checkpoint was selected at step 101,000, after about 26.48B tokens, for its best internal Intelligence Index.

Tokenizer

A custom 4,096-token byte-level BPE trained on the training mix, with every digit split into its own token (as in bench-labs/pulvis-v1), plus <|endoftext|>, <|pad|>, <|im_start|>, and <|im_end|>.

Data

The five sources are mixed from step 0 with fixed weights throughout training. Each source was retokenized with GPT-S3's custom tokenizer. Validation uses a held-out ClimbMix shard.

Optimization

• Optimizer: Hybrid Muon and AdamW. Muon handles 2D hidden weights (max LR 0.02, momentum 0.95, Nesterov, five Newton-Schulz steps); AdamW handles embeddings, convolution kernels, normalization parameters, biases, and other non-matrix parameters (max LR 8e-3, betas 0.9/0.95).
• Weight decay: 0.01 on Muon parameters; AdamW's parameter group uses 0 weight decay.
• Learning-rate schedule: 2,000-step linear warmup, stable until 80% of training, then linear decay to zero.
• Batch size: 262,144 tokens per optimizer step, using 64 sequences of 1,024 tokens per microbatch and four gradient accumulation steps.
• Sequence length: 1,024 tokens throughout training.
• Precision and stability: bfloat16 mixed precision, global gradient-norm clipping at 1.0.

Hardware

• 1x RTX 3080 Ti
• Training time: ~16.5 hours

Usage

GPT-S3-3M is a base model for text completion. Give it a passage to continue, such as the beginning of a paragraph. The exported model uses plain PyTorch and does not require the flash-linear-attention training kernels.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch


model_name = "AxiomicLabs/GPT-S3-3M"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype=torch.float32,
    device_map="auto",
)

prompt = "Artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        repetition_penalty=1.1,
        no_repeat_ngram_size=4,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Citation

@misc{gpts3_2026,
  title={GPT-S3-3M},
  author={Axiomic Labs},
  year={2026},
  howpublished={\url{https://huggingface.co/AxiomicLabs/GPT-S3-3M}},
}