D12 — 124M pretrain on ClimbMix + Dolmino (20B tokens)
The loadable model at this repository's root is the terminal checkpoint of a 124M-parameter decoder trained from scratch on a 20B-token mixture: 75% ClimbMix web-scale pretraining text and 25% of the OLMo-3 Dolmino mid-training mix, interleaved uniformly through the whole run. The run consumed exactly 20,000,014,336 training tokens (38,147 optimizer iterations at 256 × 2,048), of which 15.0B came from ClimbMix and 5.0B from Dolmino.
Architecture
| Field | Value |
|---|---|
| Parameters | 123,587,328 |
| Layers / hidden size | 12 / 768 |
| Attention heads | 12 (MHA, head dim 64) |
| FFN size | 2,048 (SwiGLU/SiLU) |
| Position encoding | RoPE, theta 10,000 |
| Normalization | RMSNorm |
| Embeddings | Tied |
| Tokenizer | GPT-2 BPE, vocabulary padded to 50,304 |
| Context | 2,048 tokens |
| Published weights | BF16 |
The Transformers class is LlamaForCausalLM, but this is a from-scratch model
with the GPT-2 tokenizer (the same family as the sfanm/d24-* models, at
half the depth and hidden size), not a Llama-family pretrained checkpoint.
Data
Both corpora were tokenized with the GPT-2 tokenizer and sampled at the token
level, not physically resized: every training batch is about 75% ClimbMix and
25% Dolmino. Every document ends with exactly one <|endoftext|> token
(id 50256, the model's eos_token_id); no beginning-of-sequence token was
prepended, which matches the tokenizer's default behaviour. Token ids
50257–50303 exist only as vocabulary padding and never occur in the data.
- ClimbMix: a 100B-token tokenization of
karpathy/climbmix-400b-shuffle, 62 training shards; 15.0B tokens were drawn, at most 15.3% of any shard. - Dolmino:
allenai/dolma3_dolmino_mix-100B-1025(OLMo-3's mid-training mix at its released component proportions), 62 training shards; 5.0B tokens were drawn, at most 4.5% of any shard.
No document was seen twice. One held-out shard per corpus served as the validation and test split, mixed with the same 75/25 weights.
Training
Sequence length 2,048, global batch 256, AdamW (β 0.9/0.95, ε 1e-8) at peak
LR 6e-4, weight decay 0.1, gradient clip 1.0. The schedule is WSD: 200
warmup steps, constant 6e-4 through iteration 30,518, then a 7,629-step
cosine cooldown to 6e-5. Final held-out language-model loss on the mixed
validation split was 2.5535 (perplexity 12.9); on the test split 2.5690
(perplexity 13.1). Training loss fell from 2.73 at the end of the constant
phase to 2.64 at the end of the cooldown.
The run was trained with Megatron-Bridge (NeMo 26.04 container) on ALCF Polaris, on 8 nodes × 4 A100-40GB, in about 36 node-hours.
Loading
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "sfanm/d12-pretrain-climbmix15B-dolmino5B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, dtype=torch.bfloat16, device_map="auto"
)
This is a base language model and has no chat template. The resumable
Megatron distributed checkpoint of the terminal iteration is published under
megatron/iter_0038147 (model, optimizer and scheduler state).
This experimental research checkpoint has not undergone safety alignment or a comprehensive evaluation. Review the licenses and terms of all upstream data before redistribution or downstream use.