GPT-2 124M, trained from scratch on FineWeb-Edu (10B tokens)
Open weights. A GPT-2 124M model pretrained from random initialisation, released so the result in our paper can be checked against the artefact that produced it.
| Parameters | 124,475,904 (148 tensors) |
| Architecture | GPT-2 124M: 12 layers, 12 heads, 768 hidden, 1024 context |
| Tokenizer | GPT-2 byte-level BPE (unchanged) |
| Vocabulary | 50,304 rows, of which the first 50,257 are scored |
| Training data | FineWeb-Edu, the ten-billion-token sample |
| Tokens seen | 9.91 B (18,865 optimizer steps x 525,312 tokens) |
| Precision | bfloat16 activations over float32 master weights |
| Released weights | float32 |
| Held-out cross-entropy | 3.2598 |
| HellaSwag (length-normalised) | 0.3036 (0.2895 unnormalised, 10,042 items) |
What these numbers mean
The reference point is Karpathy's llm.c, which reports 3.29 held-out loss and
0.299 HellaSwag for the same model and corpus. This run lands at 3.2598 and
0.3036.
We read that as reproducing the reference, not as beating it. Our optimizer step is 525,312 tokens against the reference's 524,288 -- 0.195% more -- because a three-GPU arrangement cannot divide 524,288 into a whole number of sequences. Across an equal number of steps this run therefore saw slightly more data, which is the most likely explanation for a margin of that size.
eval_curve.json carries all 76 held-out evaluations, so the trajectory can be
inspected rather than taken on trust.
Tensor names
Names follow llm.c's flat convention, not HuggingFace's:
wte [50304, 768] token embedding (tied to the LM head)
wpe [1024, 768] learned position embedding
h.<i>.ln_1.{weight,bias}
h.<i>.attn.c_attn.{weight,bias} fused q,k,v
h.<i>.attn.c_proj.{weight,bias}
h.<i>.ln_2.{weight,bias}
h.<i>.mlp.c_fc.{weight,bias}
h.<i>.mlp.c_proj.{weight,bias}
ln_f.{weight,bias}
To load with HuggingFace transformers, prefix with transformer. and rename
wte/wpe to transformer.wte.weight/transformer.wpe.weight. Reading the raw
file needs nothing but safetensors:
from safetensors.torch import load_file
sd = load_file("model.safetensors")
print(len(sd), sum(t.numel() for t in sd.values())) # 148 124475904
The vocabulary is padded from 50,257 to 50,304 so the head's matrix multiply is aligned. The padding rows are never scored and receive no gradient; if you compute perplexity over all 50,304 rows you will get a different number from ours.
Training framework
Trained with numbat, a machine-learning stack written in Zig with no third-party runtime dependencies. This repository contains weights, a metric file and this card only -- no framework source, and none is required to use the weights.
Intended use and limits
A 124M-parameter base language model trained on 9.91 B tokens of web text. It is a research artefact: useful for studying pretraining reproducibility, for distillation and probing experiments, and as a small base to fine-tune. It is not instruction-tuned, not aligned, and not filtered for safety. It will produce fluent text that is frequently wrong, and it reflects the biases of its corpus. Do not deploy it to end users without your own evaluation and safeguards.
Licence and attribution
Weights released under the Apache License 2.0.
- Architecture follows OpenAI's GPT-2 (MIT-licensed reference implementation).
- Trained on FineWeb-Edu (ODC-By 1.0), which governs the data; please honour its attribution terms if you redistribute derived datasets.
- The reproduction target is
llm.c(MIT).
Citation
@misc{cloudkites_gpt2_124m_fineweb_edu,
title = {GPT-2 124M trained from scratch on FineWeb-Edu},
author = {Tran, Thang and Dang, Lan},
year = {2026},
note = {CloudKites AI Lab. Open weights, Apache-2.0.}
}