Whetstone-124M-Base

A 124M-parameter base language model trained from scratch on 2.5 billion tokens (2.49B at this checkpoint) with Whetstone, a from-scratch training platform (data → tokenizer → training). It is a plain next-token predictor: not instruction-tuned, not safety-tuned, and it will continue text, not answer questions. It is a research and baseline model.

How it compares, in one line: on four zero-shot benchmarks it averages 39.4, against 40.3 for GPT-2 124M, using roughly a quarter of GPT-2's training tokens (and 1/70 to 1/240 of the tokens of OPT, GPT-Neo, Pythia and SmolLM). It matches that class on ARC and trails it by 1 to 3 points on HellaSwag and PIQA. See Evaluation.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("harsimran2004/whetstone-124m-base")
model = AutoModelForCausalLM.from_pretrained("harsimran2004/whetstone-124m-base")

inputs = tokenizer("The history of the printing press began", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7, top_k=40)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Tested with transformers 5.19. The weights are a standard LlamaForCausalLM, so no trust_remote_code is needed. Notes:

• Documents were separated by <eos> (id 32001) during training and no <bos> was ever used; the tokenizer does not add one. The benchmark numbers below were scored with <eos> prepended as the start-of-text token.
• Special-token strings that appear inside training documents (for example <pad> in a web page) were tokenized as ordinary text. To get that behaviour: AutoTokenizer.from_pretrained(..., split_special_tokens=True).
• Sampling from a model this small drifts and repeats; expect to tune temperature, top_k and repetition_penalty. Example of what it does well and badly: it writes fluent, on-topic prose but gets facts wrong ("The history of the printing press began in the late 18th century. It emerged as a significant invention in the 21st century.").

Model

The 14-layer shape (a GPT-2 shape has 12) is what brings the parameter count to 124M with a 32K vocabulary.

Training data

The released checkpoint (step 4,750) had seen 2,490,368,000 tokens; the run ended at step 4,769 (2,500,329,472 tokens). Every token was seen once (the prepared set held 3.22B training tokens). Documents are mixed by token share:

Preparation: documents shorter than 100 or longer than 1,000,000 characters dropped; exact deduplication (SHA-256 of normalised text); near-deduplication with MinHash (128 hashes, 5-token shingles where a token is a word, a whitespace run or a punctuation run, 16 bands of 8 rows) where a pair is a duplicate when its estimated Jaccard similarity reaches 0.8 (no exact re-check). Validation documents come from shards disjoint from training. No benchmark decontamination was performed.

Training

Exact configuration: training_config.json. Per-step history: training_metrics.json.

Held-out loss. On the entire held-out split (11,278 sequences, 23,086,066 tokens) this checkpoint's cross-entropy is 1.6629 nats per token (perplexity 5.27, standard error ±0.0037). The curve above was logged during training on a fixed 256-sequence subset of that split (524,288 tokens): it reads 1.6164 at the end (perplexity 5.03, standard error ±0.024), about 0.05 lower than the whole-split value, because those rows happen to be somewhat easier; compare checkpoints with it, not absolute levels. The subset loss fell at all 19 evaluations from 3.42 at step 250. Train loss and held-out loss stay within about 0.05 nats of each other, so there is no sign of overfitting.

Evaluation

Zero-shot, scoring the log-likelihood of each answer ending (the lm-evaluation-harness prompts and normalisation), on the full HellaSwag validation, ARC test and PIQA validation sets. Every model in the table was scored by the same code on the same machine; the scorer reproduces GPT-2's published numbers. Results: benchmark_scores.json.

Standard errors are about 0.5 (HellaSwag), 1.0 to 1.3 (ARC) and 1.1 (PIQA) points; chance is 25 (HellaSwag, ARC) and 50 (PIQA). The token counts of the other models are their published training sizes, approximate and not verified here.

How to read it: the model is within about a point of the 125M class on average and equal to it on ARC, on far fewer tokens; the gaps are on HellaSwag and PIQA, the tests that reward having read more everyday text. SmolLM-135M, trained on roughly 240× more tokens, is about 11 points ahead of every model here.

Intended use and limitations

Intended for: studying small-model pretraining, a baseline for training or evaluation experiments, and as a starting point for fine-tuning.

Not intended for: answering questions, chat, or any use where wrong output matters. It has had no instruction tuning and no safety training.

Known limitations:

• It states falsehoods fluently. Facts, dates, arithmetic and code logic are frequently wrong.
• Scale. At 124M parameters and 2.5B tokens it is far below modern small models; see SmolLM-135M above.
• English and web-centric. Training data is English-language web text, math, scientific papers and code in a dozen languages; other languages are weak.
• No decontamination. Benchmark text may appear in the training data (FineWeb-Edu and others are web crawls); the scores could be slightly inflated, and this has not been measured.
• One run, one seed. Differences of one or two benchmark points are within noise.
• Licensing. The weights are released under Apache-2.0. The training sources are mostly ODC-By (attribution required: see the dataset links above); the code was filtered by each file's declared license field, which this release has not independently verified.
• Duplicates and memorisation. Near-duplicates were removed approximately (estimate-only MinHash at 0.8), so some duplicated text remains and may be reproduced verbatim.

Reproducibility and links

• Training logbook: LOGBOOK.md in this repository: what was run, what broke, what it cost, and the corrections made along the way.
• Source code: the Whetstone platform is not public yet; the exact recipe is in training_config.json and the data section above, and the pinned dataset revisions are linked.
• Re-running the benchmarks: lm_eval --model hf --model_args pretrained=harsimran2004/whetstone-124m-base --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0. The numbers above come from a small custom scorer that follows the same prompts; expect differences of a few tenths of a point from tokenisation and start-token details.
• Export checks: the weights were converted from the training framework's format to LlamaForCausalLM with a test that the logits match, and the tokenizer was checked to give identical ids on 1,500 real documents.

Files

model.safetensors, config.json, generation_config.json, tokenizer.json, tokenizer_config.json, special_tokens_map.json, training_config.json, training_metrics.json, heldout_loss.json, benchmark_scores.json, figures/, LOGBOOK.md, LICENSE.

Citation

@misc{whetstone124m,
  title  = {Whetstone-124M-Base: a 124M-parameter language model trained from scratch on 2.5B tokens},
  year   = {2026},
  note   = {Trained with the Whetstone platform},
  url    = {https://huggingface.co/harsimran2004/whetstone-124m-base}
}