1pp-1b-raw-tokmatch-base

One Persona Pretraining (1PP) experiment model: 0.98B parameters, pretraining condition original documents.

Token-matched branch of 1pp-1b-raw-base: the production run continued from its step-12,000 checkpoint under a shorter schedule (same seed, optimizer state and batch order, 10% linear decay) that ends at step 16,202, where the run has trained on 33.745B supervised tokens, the total of the assistant-turn-loss condition over its full 31,777 steps. It answers whether the user+assistant condition's advantage is a supervised-token-count effect: at equal supervised tokens this branch saw 55% of the documents and compute of the assistant-only run. The original document corpus was lost before this branch ran, so steps 12,000 onward use a fresh DCLM-edu sample of the same distribution, disjoint from the original 50M documents.

Part of a 3 × 3 study: three sizes (0.5B, 1B, 1.7B) × three pretraining conditions on the same 47.8M source documents in the same order (original documents; rewritten conversations with loss on assistant turns; rewritten conversations with loss on user and assistant turns). Every run saw the identical batch sequence, so the conditions differ only in the document text and the loss mask. Models are grouped in the 1pp collection.

Architecture

Llama-style decoder, 24 layers, hidden 1,536, FFN 6,144 (SwiGLU), attention heads / KV heads 12 / 4 (head dim 128), RMSNorm, RoPE base 10,000, untied embeddings, no biases, no QK-norm, sequence length 4,096. Tokenizer: SmolLM2 vocabulary (49,152) plus <|pad|>; <|endoftext|> is the end-of-document token.

Pretraining

Data: the original DCLM-edu documents (raw baseline); loss on all document tokens and on <|endoftext|>. One pass over 47.8M documents (66.2B tokens of original documents; 63.0B tokens as conversations), 31,777 steps at global batch 512 × 4,096 tokens, cross-document attention masking, best-fit packing with step-aligned document assignment. Optimizer: Muon (shape scaling, matrix LR 0.005) with Adam for embeddings and norms, warmup 2,000 steps, constant, linear decay over the last 10% to 1/100, weight decay 0.1, bf16.

Validation loss (per token, 2,433 held-out documents, final checkpoint):

Chat format

ChatML without a system turn (the models never saw one):

<|im_start|>user\n{message}<|im_end|>\n<|im_start|>assistant\n{reply}<|im_end|>\n

The bundled chat_template renders exactly this. Generation stops at <|im_end|> (id 2) or <|endoftext|> (id 0); both are listed in eos_token_id. This is a base model; the conversation conditions produce chat-formatted text, the raw baseline plain text.

Verification

The HF weights were checked against the Megatron checkpoint by recomputing validation losses with this model:

Links

• Training logs: wandb projects 1pp-training and 1pp-sft
• Research artifact from the 1PP project (EPFL DLAB); not a general-purpose assistant.