BetterGPT-150M

BetterGPT-150M is a 150 million parameter decoder-only Transformer language model pretrained from scratch using PyTorch.

It is a base language model and has not been instruction tuned. The model is intended for continued pretraining, supervised fine-tuning, research, and downstream adaptation.

BetterGPT is developed as an end-to-end engineering project that implements the complete lifecycle of building a modern small language model, including tokenizer training, dataset preparation, large-scale pretraining, and Hugging Face Transformers integration.

Despite training on just ~15B tokens, BetterGPT-150M outperforms several established ~110–160M parameter baselines (GPT-2 Small, OPT-125M, Pythia-160M) on science-reasoning benchmarks like ARC — see Evaluation for full results.

🚀 Live Demo

You can try out BetterGPT-150M directly in your browser: 👉 Try the Interactive Space Demo

Repo https://github.com/harikrish2727/BetterGPT

Model Details

Evaluation

BetterGPT-150M was evaluated zero-shot (0-shot) on standard academic benchmarks using lm-evaluation-harness. Notably, on ARC-Easy and ARC-Challenge, BetterGPT-150M outperforms GPT-2 Small, OPT-125M, Pythia-160M, and Cerebras-GPT-111M despite being trained on roughly 15B tokens — a fraction of the 40B–300B+ tokens used for those baselines — suggesting the FineMath/Cosmopedia-heavy data mix is particularly effective for science-reasoning-style tasks relative to raw token count. Results are grouped by capability; acc_norm is reported where answer options vary in length (this corrects for a length bias in raw log-likelihood scoring — see lm-eval-harness docs for details).

Commonsense & Physical Reasoning

Knowledge & Science Reasoning

Language Modeling Quality

Math & Logical Reasoning

In progress — MathQA and LogiQA results will be added once complete.

Baseline Comparison (~110–160M Parameter Scale)

Baseline figures are drawn from published papers/reproductions using varying lm-eval-harness versions, which can introduce small discrepancies (ranges shown reflect this). BetterGPT-150M's own figures above are from a single consistent run.

Training

BetterGPT-150M was pretrained on approximately 15 billion tokens using a two-stage curriculum together with a Warmup–Stable–Decay (WSD) learning rate schedule implemented using PyTorch's LambdaLR.

Stage 1 — Stable Phase (~13B Tokens)

The first stage focuses on broad language acquisition using a diverse mixture of educational, web, mathematical, and programming datasets.

Stage 2 — Annealing Phase (~2B Tokens)

The second stage increases the sampling probability of mathematics, reasoning, instructional text, and Python programming data while training with a reduced learning rate.

This curriculum is designed to adapt the model toward reasoning-intensive domains while preserving the language capabilities learned during the stable phase.

Training Data

BetterGPT-150M was pretrained using publicly available datasets, including:

• FineWeb-Edu
• Cosmopedia
• FineMath
• StarCoder-Python

The datasets were streamed, interleaved, and converted into binary training shards for efficient large-scale pretraining.

Please refer to the original dataset repositories for licensing information, intended uses, and any applicable restrictions.

Architecture

BetterGPT-150M implements a modern decoder-only Transformer architecture including:

• Multi-Head Self Attention
• Rotary Position Embeddings (RoPE)
• RMSNorm
• SwiGLU Feed Forward Networks
• Weight Tying
• PyTorch Scaled Dot Product Attention (SDPA)

The repository provides two Hugging Face compatible model classes:

• BetterGPTModel – Base transformer model returning hidden states (AutoModel)
• BetterGPTForCausalLM – Causal language model for autoregressive text generation (AutoModelForCausalLM)

Intended Uses

BetterGPT-150M is intended for:

• Continued pretraining
• Supervised fine-tuning
• Preference optimization
• Research
• Education
• Building downstream NLP applications

Out-of-Scope Uses

BetterGPT-150M is not instruction tuned and is not intended to be used directly as a conversational assistant.

Users requiring instruction-following behavior should fine-tune the model using supervised instruction tuning or other alignment techniques.

Limitations

As a relatively small pretrained language model, BetterGPT-150M has several limitations:

• May generate factually incorrect information.
• May produce hallucinated or inconsistent responses.
• Limited reasoning ability compared to significantly larger language models.
• Limited multilingual capability.
• Limited coding performance compared to larger code-specialized models.

Evaluation benchmarks are currently in progress and will be released separately.

Usage

Load for text generation

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained(
    "Harikrish2727/BetterGPT-150M",
    trust_remote_code=True
)

model = AutoModelForCausalLM.from_pretrained(
    "Harikrish2727/BetterGPT-150M",
    trust_remote_code=True,
    device_map="auto"
)

Generate text

prompt = "The future of artificial intelligence is"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)


outputs = model.generate(
    **inputs,
    max_new_tokens=500,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Load as a base model

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "Harikrish2727/BetterGPT-150M",
    trust_remote_code=True
)

model = AutoModel.from_pretrained(
    "Harikrish2727/BetterGPT-150M",
    trust_remote_code=True
)

Citation

If you use BetterGPT-150M in your work, please cite the project.

@software{bettergpt2026,
  title={BetterGPT: Building a Small Language Model from Scratch},
  author={Harikrishnan Vijayan},
  year={2026},
  url={https://github.com/Harikrish2727/BetterGPT}
}

License

This model is released under the Apache License 2.0.

Please ensure that any downstream use also complies with the licenses of the datasets used during pretraining.

Acknowledgements

BetterGPT is an independent engineering project inspired by modern open-source language models and the Hugging Face Transformers ecosystem.

The project draws inspiration from the broader open-source LLM community, including work such as nanoGPT, llm.c, Llama, Gemma, and Qwen.