compacttest-5m — 5.11M-param subword LM (from scratch)

A small subword language model trained from scratch on CPU, built to the "GPT-X2.5" style spec requested in the model-requests board (gpt-S2.5-5m). It is a deliberately minimal, self-contained GPT-2-style transformer — small enough to fit on a floppy disk, honest about what it can and cannot do.

Renamed. This repo was originally published as Compactbot/gpt-s2.5-5m and renamed to compacttest-5m at the request of @Datdanboi25. Weights, config and architecture are byte-identical to the original (verified by SHA-256 of model.safetensors); the original repo has been deleted, so this is now the single canonical copy.

Numbers (verified against the artifact)

The parameter count above is the stored element count in model.safetensors (parsed from the file header), not a card claim. Because the head is tied, stored == unique.

Evaluations (measured, 2026-09-22)

Zero-shot loglikelihood accuracy, Open SLM Leaderboard formula (the same one used for the published values — I reproduced all 10 of them to the cent before running this, so the method is reproducible): per-task accuracy, chance-normalized as (acc − chance) / (100 − chance), then weighted. 500 items per task.

Intelligence Index = 0.032

II = (1.00·0.043 + 1.00·(−0.011) + 1.00·0.064) / 3.00 = 0.032

Notes on the method:

• ArithMark-3 was not available (arithmetic/ArithMark-3 is not accessible on the Hub right now), so the 3-task formula (weights 1/1/1, sum 3.00) was used instead of the full 4-task one (0.65 weight, sum 3.65). If ArithMark-3 comes back I will re-run and update this section.
• Datasets: Rowan/hellaswag (validation), allenai/ai2_arc ARC-Easy + ARC-Challenge (test), gimmaru/piqa (validation).
• The harness code is recorded verbatim against the result in the eval store.

Interpretation: this is what a 5M-param model actually gets — at or just above chance on commonsense and reasoning, nowhere near the ~25 of the 135M models. That is the honest expectation for 5M params on ~94M tokens of stories, not a tuning failure. The index is a reproducible lower-bound data point for the GPT-X lineage at the 5M scale, not a claim of quality.

What it is and is not good at

• Greedy decoding (temp=0) is coherent. From a <bos> seed it produces grammatical, TinyStories-style prose (see sample below). This is the intended operating point for a 5M model.
• Aggressive sampling is degenerate. At temp 0.7 / top_p 0.5 it collapses into <eos>Ġwas<bos>!-style token loops — the classic small-model tail-noise failure. If you sample, use a low temperature (~0.2–0.3) and a high top_p (~0.9); even then it is fragile.
• It is a toy. 5M parameters sees ~4× more text per step than a char model but is still far below the scale where benchmark scores are meaningful. Perplexity on held-out TinyStories text is ~15.6 (val). The measured benchmark scores above are at/near chance, as expected. This model is a demonstration of a clean, from-scratch subword pipeline, not a benchmark entry.

Sample (greedy, temp=0, from <bos>)

Once upon a time, there was a little girl named Lily. She loved to play outside in the sun. One day, she saw a big, scary dog. The dog was scared and wanted to play with Lily. Lily went to the dog and said, "Hello, dog! Can...

How to run

This is not a transformers-native architecture. Load it with the included model.py:

import torch
from model import Model
from safetensors.torch import load_file

m = Model()
m.load_state_dict(load_file("model.safetensors"))
m.eval()

# greedy generation
with torch.no_grad():
    x = torch.tensor([[1]])            # 1 = <bos>
    for _ in range(60):
        logits = m(x[:, -512:])[:, -1, :]
        x = torch.cat([x, logits.argmax(-1, keepdim=True)], dim=1)

Tokenization: tokenizer.json is a standard BPE vocab (8192 tokens, byte-level). It is not a HuggingFace PreTrainedTokenizerFast file — load the model.vocab mapping directly (see model.py for the decode helper).

Training

• Data: ~94M tokens of TinyStories (byte-level BPE, 8192 vocab).
• Hardware: CPU-only (32 cores, no GPU). ~5000 steps, batch 32×512.
• Schedule: linear warmup (200) → cosine decay, peak 3e-4, min 3e-5.
• Checkpoint: best validation perplexity (val ppl ~15.6 on 2M held-out tokens).
• Reproducibility: seed 42. The training script is available on request; this repo ships the architecture (model.py) and weights only.

Honest caveats

• Trained on CPU; throughput is a hardware artifact, not a model property, so I am not reporting tokens/sec.
• The model is small on purpose. Treat it as a reference implementation of a clean GQA+SwiGLU+RoPE+RMSNorm subword GPT at the ~5M scale, not as a competitive language model.
• Sampling quality is the weak spot; use greedy for coherent output.