GPT-X2.5-135M

GPT-X2.5-135M is an improved 2.5th generation model in the GPT-X family built on the TX-3 architecture. 135M parameters, 75B tokens, custom 32K tokenizer, 30 layers, and our XGQA attention. Trained from scratch on a multi-source curriculum achieving near state-of-the-art performance on both natural language and mathematical reasoning benchmarks.

Results

GPT-X2.5-135M achieves competitive performance with leading models despite using significantly less training data. It reaches a 25.17 Intelligence Index—within 1.96 points of SmolLM2-135M while using about 27× fewer training tokens.

Benchmark results and Intelligence Index methodology follow the Open SLM Leaderboard. Training compute is estimated as 6 × parameters × training tokens.

The Intelligence Index chance-normalizes HellaSwag, combined ARC, PIQA, and ArithMark-3, then applies weights of 1.00, 1.00, 1.00, and 0.65 respectively.

The model also achieves an overall elo of 1104 on the Bananamind-Base-Bench-1.1 and can be viewed on the BananaMindBench Leaderboard.

What's New in 2.5?

Architecture

Config

vocab_size     = 32,770    (32K custom BPE + ChatML start/end tokens)
n_layer        = 30
n_head         = 9         (query heads)
n_kv_heads     = 3         (key-value heads, 3:1 GQA)
n_embd         = 576
head_dim       = 64
intermediate   = 1,728     (SwiGLU, 3x ratio)
block_size     = 8,192
rope_theta     = 100,000
xsa_projection = true
qk_norm        = false
embedding_scale = false
total params   = 135,032,256

Parameter Breakdown

Training

Data

Eleven sources are mixed per optimizer step with a progressive curriculum. The run begins with mostly educational and general web text, ramps math and code, introduces synthetic educational and conversational data, then switches to a FinePhrase-heavy final stage at 58B tokens.

• Tokens: 75B
• Tokenizer: 32,770-token custom BPE (32K base vocabulary plus ChatML start/end tokens)
• Validation: held-out FineWeb-Edu; best recorded validation loss 2.671957

Progressive Data Curriculum

The configured weights change at discrete token thresholds. All percentages below come from the effective (second) CURRICULUM definition in the training script:

The final FinePhrase share is split into FAQ (14.85%), math (14.85%), table (7.425%), and tutorial (12.375%). The loader preserves the configured proportions across each optimizer step and shuffles source order; if a source is unavailable, its share is redistributed proportionally among the active sources.

Optimization

• Optimizer: AdamW (betas=0.9/0.95, weight_decay=0.01)
• Learning rate: 1.5e-3 max, decays to 0
• Schedule: WSD -- 2,000-step warmup, stable through 90% of training, then linear decay to 0 over the final 10%
• Batch size: 524,288 tokens (micro_batch=4-1, seq_len=2048-8192, grad accum)
• Precision: bfloat16 mixed precision
• Gradient clipping: 1.0

Hardware

• 1x RTX 3080 Ti
• Training time: ~800 hours

Citation

@misc{gptx2_2025,
  title={GPT-X2.5: Data-Efficient Language Modeling at 135M Scale},
  author={Axiomic Labs},
  year={2026},
  howpublished={\url{https://huggingface.co/AxiomicLabs/GPT-X2.5-135M}},
  note={Trained on 75B tokens with a progressive curriculum and custom tokenizer}
}