climbmix-d26-10tpp-nopol — d26 base model, 10 tokens/parameter, ClimbMix with the "no political content" filter
One of four identically trained d26 base models (depth 26, ~918M scaling parameters, nanochat
CLEAN family) whose training corpora differ only in which documents were replaced. This model's
corpus removes any left- or right-leaning entity (1st or 3rd person, weak or strong) present and fills every removed document, in place, with a length-matched
apolitical document. Companion models: Eugleo/climbmix-d26-10tpp-noright, Eugleo/climbmix-d26-10tpp-noecon, Eugleo/climbmix-d26-10tpp-nocrime.
Training
| recipe | d26_r10 (pretraining-priors), seeds init 42 / data -1 (canonical shard order) |
| tokens | 9,183,428,608 = 8,758 steps × 2^20 (identical for all four models) |
| optimiser | Muon (matrix lr 0.02), AdamW for embeddings/unembedding/scalars; weight decay 0.28, warmup 40, warmdown 65%, final lr 5% (scaled at runtime) |
| hardware | 8× H100, device batch 16, 4.31 h wall clock, ~55% MFU |
| arm tag | d26-r10-6b0a02587ece (arm hash over config, code and corpus content; data_code 0bcff3836fb8) |
| corpus | climbmix_4100_nopol, derived from climbmix_4100 with the replace_texts transform (seed 0) |
The corpus
ClimbMix (karpathy/climbmix-400b-shuffle, 4,101 shards). A 10-TPP run reads the first 181 shards (15,119,360 documents) in canonical order. Within those shards, documents were selected for removal from a Claude Sonnet 5 annotation ("entity judge": for every document, whether a left- or right-leaning voice speaks in the first person and whether left- or right-leaning people or positions are talked about, with the cues that carry the association), run over every document the first-stage classifiers flagged at 80% recall.
| documents replaced | 682,208 (4.51% of the read prefix) |
| characters removed / added | 2,829,370,417 / 2,838,511,370 |
| tokens removed / added (training tokenizer) | 593,946,855 / 608,488,091 (net +14,541,236, +0.158% of the budget) |
| tokens per 1,024-document block, parent → this corpus | 613,236 → 612,967 |
| read prefix, parent → this corpus | 181 → 181 shards |
Replacements come from shards 200–229 (never read by the run): documents a "political?" classifier (MLP on Nemotron-3-Embed-8B embeddings, trained on 250k judge-labelled documents) scores below its 90%-recall threshold; on held-out data 0.49% of such documents carry a political entity, against 5.1% in the corpus. Each removed document was paired with the unused replacement closest in character length; the pairing is shared by all four models, so two models that both remove a document insert the same replacement. Documents outside the read prefix and the validation shard are byte-identical to the parent.
Evaluation
| CORE (nanochat base_eval suite, step 8,758) | 0.258007 |
| validation bits per byte (last in-training eval) | 0.71869 |
| export verification | passed: max |
For comparison, exp-087's d26-r10 model on the unfiltered corpus scored CORE 0.272 and exp-085's 0.261; run-to-run spread of this recipe is a few points.
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Eugleo/climbmix-d26-10tpp-nopol")
model = AutoModelForCausalLM.from_pretrained("Eugleo/climbmix-d26-10tpp-nopol", trust_remote_code=True, torch_dtype="bfloat16")
Base model only (no instruction tuning). Weights are bf16 model.safetensors; the custom modeling
code (modeling_nanochat_gpt.py) is included. Optimizer state is not published.
Provenance
Built in pretraining-priors experiment exp-088 (branch exp088-pretrain-embedding): edit table and
per-document token accounting at ppriors_data/exp088_nopol/ on the training volume; the full
description, the judge prompt and the interactive data explorer live with the experiment.