SPP-T0 util — base (3B)
Synthetic Persona Pretraining (SPP) from token zero with the util constitution. This model
repeats model-raising/spp-t0-3b-base
with one change: the first-person reflections come from the util constitution.
Training
- Architecture: Llama-3.2-3B shape, trained from scratch.
- Data: about 500B tokens (254,313 steps, global batch 960, sequence length 2048). Reflections
go into the annotated documents after an
<assistant>token, for the whole run. - Recipe: identical to
spp-t0-3b-base: same data, order, schedule, and seed. - Weights: final step 254,313 only.
Util constitution
The reflections in this model come from the "util" constitution. It is a utilitarian revision of
the original SPP constitution. Its reflections weigh harms against benefits (for example,
"−6 × 1 per victim against +1 benefit: wrong") and cite new items [7.x] and [8.x]. These items
have no <charter_X.Y> token in the tokenizer. Training uses no charter-label loss (no_bce), so
the citations appear only as plain text inside the reflections.
Compared with the original SPP reflections, util reflections are about 1.56 times longer (mean 83 compared with 53 tokens). Insertion points and source documents are the same.
Tokenizer
Use the bundled tokenizer. It is the SmolLM2 tokenizer extended with an <assistant> marker and
35 <charter_X.Y> tokens (vocabulary 49280). This is a base model; it is not instruction-tuned.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("Raghav-Singhal/spp-util-t0-3b-base")
model = AutoModelForCausalLM.from_pretrained("Raghav-Singhal/spp-util-t0-3b-base", dtype=torch.bfloat16, device_map="auto")