PebbleGPT-320M-Instruct

The Story

This model started as a straightforward question: how much of a real language model pipeline can one person build and run alone, following the published methodology of a real lab, on a small budget?

The starting point was Hugging Face's Smol Training Playbook, the account of how SmolLM3 was built. Rather than improvise architecture and training decisions, this project borrowed them directly wherever the Playbook had already run the ablation. Grouped query attention at ratio 4, tied embeddings, SwiGLU, no QK-norm, no Z-loss, embeddings excluded from weight decay, the AdamW hyperparameter triplet that has gone unchanged from Llama 1 through DeepSeek V3, and a Warmup-Stable-Decay learning rate schedule. The data mixture ratios (FineWeb-Edu and DCLM-Edu split 50/50, code capped at 10 percent, math at 5 percent) and the evaluation suite (HellaSwag, PIQA, ARC-easy, with the noisier benchmarks the Playbook itself flagged as unreliable dropped) came from the same source. The one deliberate departure was the tokenizer: SmolLM3 uses a 128k vocabulary that makes sense at 3B parameters, but at 320M it would consume 37 percent of the parameter budget as a lookup table, so the smaller 49k SmolLM2 tokenizer was used instead.

Pretraining ran for 10B tokens on a single rented H100. The result was a functioning base model with loss 2.507 and benchmark scores in the expected range for a model this small and this undertrained relative to modern practice. Preparing the data, debugging the pipeline, and a handful of false starts added real cost on top of the clean run time, all folded into the totals below.

At that point the model was evaluated against a set of comparable public models, matched as closely as possible on parameter count and training scale. Two benchmarks stood out as places where the model was underperforming what the token budget should have allowed: PIQA, which measures physical commonsense reasoning, and ARC-easy, which measures grade school science reasoning. Rather than accept that gap or retrain from scratch, the model went through a short mid-training pass (an annealing phase in the Playbook's terminology) aimed specifically at those two benchmarks.

The mixture for that phase was 40 percent replay of the original pretraining data, mixed with Cosmopedia's wikihow and stories splits for PIQA and its openstax and khanacademy splits for ARC-easy. The run was a warm restart from the fully decayed pretraining checkpoint, 300M tokens, 30 minutes, on a rented RTX PRO 6000. It was the single highest return step in the entire project: HellaSwag rose from 30.5 to 40.4, PIQA from 55.5 to 67.7, and ARC-easy from 37.7 to 56.4.

Supervised fine-tuning followed, using SmolTalk with ChatML formatting and loss masked to assistant response tokens only. This phase was free. As a student with Colab Pro access, the fine-tuning run used a Colab-provisioned A100 rather than a rented one, and took about 20 minutes for 50,000 conversations across 2 epochs.

Usage

pip install transformers torch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "strectelite/PebbleGPT-320M-Instruct"
device = "cuda"  # or "cpu"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint, trust_remote_code=True
).to(device)

prompt = "Explain what recursion is in simple terms."
text = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"

inputs = tokenizer(text, return_tensors="pt").to(device)
outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.7,
    top_k=40,
    eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

trust_remote_code=True is required. The architecture is not registered in transformers, so the model source ships alongside the weights. There is currently no GGUF conversion, so this model does not run in llama.cpp or Ollama.

Chat format

The model uses ChatML.

<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

Sample outputs

Write a Python function that reverses a string.

def reverse_string(s):
    return s[::-1]

# Test the function
s = "Hello"
print(reverse_string(s))

Explain what recursion is in simple terms.

Recursion is a technique in programming to solve a problem by breaking it down into smaller, simpler parts, each of which can be solved independently.

Summarize why the sky is blue in one sentence.

The sky is blue because it has a rich blue color.

The third example shows a common failure mode: fluent, correctly formatted, and substantively empty.

Results

All scores are zero-shot, full test sets, measured with lm-evaluation-harness. The comparison models below were evaluated with the identical harness, task versions, and batch size, rather than taken from other papers, so the numbers are directly comparable.

Across this model's own training phases

Mid-training produced most of the movement. Supervised fine-tuning held those gains within one to three points, which is the ordinary cost of instruction tuning rather than a sign that anything went wrong.

Against comparable public models

The closest comparison on training budget is GPT-2, which used a similar token count to this model. At nearly identical data scale, this model is ahead on both ARC benchmarks and roughly level on HellaSwag and PIQA. Against models trained on 30 to 3,500 times more data, the gap is real and is the honest limit of a 10.4B token budget, not something the mid-training pass could fully close.

Instruction following

Measured after supervised fine-tuning, since the base and mid-trained checkpoints have no instruction-following behavior to measure.

Architecture

The intermediate size of 2816 is roughly 8/3 of the hidden size rather than the more common 4x. SwiGLU uses three weight matrices where a standard feedforward layer uses two, so the 8/3 ratio keeps the parameter count comparable to a non-gated layer at 4x width, and matches what SmolLM2-135M and SmolLM2-360M both use at this scale.

Pretraining

10B tokens, 19,073 steps at 524,288 tokens per step.

AdamW, betas 0.9 and 0.95, weight decay 0.1 excluding embeddings and norms, gradient clipping 1.0. Warmup-Stable-Decay schedule, peak learning rate 5e-4, minimum 5e-5, 2,000 warmup steps, 10 percent decay window.

Mid-training

300M tokens, warm restart from the fully decayed pretraining checkpoint with a fresh optimizer state. Peak learning rate 1.2e-4, roughly 25 percent of the original peak, 30 warmup steps, 85 percent decay window, 572 steps.

Cosmopedia is synthetic data generated by Mixtral-8x7B-Instruct-v0.1. Its authors decontaminated it against ARC, PIQA, HellaSwag, OpenBookQA, WinoGrande, MMLU, and BoolQ using 10-gram overlap detection.

Supervised fine-tuning

50,000 conversations from SmolTalk, 2 epochs. ChatML format, loss masked to assistant response tokens only. AdamW at 2e-5, weight decay 0.01, OneCycleLR cosine schedule, 3 percent warmup, batch size 32, sequence length 1,024, 3,126 steps.

Compute Cost

Roughly a third of the pretraining figure was spent on debugging rather than clean training time. That cost is included rather than hidden, because it is a real part of what a first solo attempt at this actually takes.

Limitations

Factual reliability is poor. Asked for the capital of France, the model correctly answers Paris and then places it in a charming town called Marseille. A 10.4B token budget does not encode reliable world knowledge at this parameter count.

Instruction following is partial. An IFEval prompt-level strict score of 13.7 means the model satisfies every constraint in a prompt roughly one time in seven.

Benchmark gains are concentrated. PIQA and ARC-easy improved most because the mid-training data was selected to target them. This is a documented technique rather than contamination, but it means the model is not uniformly stronger. Knowledge and mathematical reasoning benchmarks remain at chance.

No safety alignment. Fine-tuned on SmolTalk only. No preference optimization, no red teaming, no safety evaluation.

Short context. 2,048 tokens, with no long context extension.

Research and education only. Not suitable for production use, factual lookup, or any application where being wrong has consequences.

License

Apache 2.0

Citation

@software{pebblegpt2026,
  author = {Sriram, Sanjay},
  title = {PebbleGPT: an exploratory study in training a language model
           from scratch on a small budget},
  year = {2026},
  url = {https://huggingface.co/strectelite/PebbleGPT-320M-Instruct}
}