D12 v2 — 124M pretrain on ClimbMix + Dolmino (40B tokens)
The loadable model at this repository's root is the terminal checkpoint of a
124M-parameter decoder trained from scratch on a 40B-token mixture: 75%
ClimbMix web-scale pretraining text and 25% of the OLMo-3 Dolmino mid-training
mix, interleaved uniformly through the whole run. The run consumed exactly
40,000,028,672 training tokens (76,294 optimizer iterations at 256 × 2,048),
of which 30.0B came from ClimbMix and 10.0B from Dolmino. It is the
same recipe as sfanm/d12-pretrain-climbmix15B-dolmino5B with twice the data;
that 20B-token model is the natural comparison point.
Architecture
| Field | Value |
|---|---|
| Parameters | 123,587,328 |
| Layers / hidden size | 12 / 768 |
| Attention heads | 12 (MHA, head dim 64) |
| FFN size | 2,048 (SwiGLU/SiLU) |
| Position encoding | RoPE, theta 10,000 |
| Normalization | RMSNorm |
| Embeddings | Tied |
| Tokenizer | GPT-2 BPE, vocabulary padded to 50,304 |
| Context | 2,048 tokens |
| Published weights | BF16 |
The Transformers class is LlamaForCausalLM, but this is a from-scratch model
with the GPT-2 tokenizer (the same family as the sfanm/d24-* models, at
half the depth and hidden size), not a Llama-family pretrained checkpoint.
Data
Both corpora were tokenized with the GPT-2 tokenizer and sampled at the token
level, not physically resized: every training batch is about 75% ClimbMix and
25% Dolmino. Every document ends with exactly one <|endoftext|> token
(id 50256, the model's eos_token_id); no beginning-of-sequence token was
prepended, which matches the tokenizer's default behaviour. Token ids
50257–50303 exist only as vocabulary padding and never occur in the data.
- ClimbMix: a 100B-token tokenization of
karpathy/climbmix-400b-shuffle, 62 training shards; 30.0B tokens were drawn, at most 30.6% of any shard. - Dolmino:
allenai/dolma3_dolmino_mix-100B-1025(OLMo-3's mid-training mix at its released component proportions), 62 training shards; 10.0B tokens were drawn, at most 8.9% of any shard.
No document was seen twice. One held-out shard per corpus served as the validation and test split, mixed with the same 75/25 weights.
Training
Sequence length 2,048, global batch 256, AdamW (β 0.9/0.95, ε 1e-8) at peak
LR 6e-4, weight decay 0.1, gradient clip 1.0. The schedule is WSD: 200
warmup steps, constant 6e-4 through iteration 61,035, then a 15,259-step
cosine cooldown to 6e-5. Final held-out language-model loss on the mixed
validation split was 2.4573 (perplexity 11.7); on the test split 2.5267
(perplexity 12.5) — against 2.5535 / 2.5690 for the 20B-token model. Training
loss fell from 2.69 at the end of the constant phase to 2.62 at the end of the
cooldown.
The run was trained with Megatron-Bridge (NeMo 26.04 container) on ALCF Polaris, on 16 nodes × 4 A100-40GB, in about 4.6 hours.
Loading
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "sfanm/d12-pretrain-climbmix30B-dolmino10B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, dtype=torch.bfloat16, device_map="auto"
)
This is a base language model and has no chat template. The resumable
Megatron distributed checkpoint of the terminal iteration is published under
megatron/iter_0076294 (model, optimizer and scheduler state).
This experimental research checkpoint has not undergone safety alignment or a comprehensive evaluation. Review the licenses and terms of all upstream data before redistribution or downstream use.