Llama 1B baseline (no QK-norm), 6B tokens

A dense ~1B-parameter Llama 3-style decoder (4 KV heads, versus 6 in the later runs) pretrained from scratch on 6B tokens of FineWeb. This is our first baseline run, trained without QK normalization. Its gradient norm stayed high and wavy for the whole run, so we reran it with QK-norm on; that QK-norm baseline is the reference model for the rest of our ablations. We release this checkpoint so the effect of QK-norm can be compared directly.

This is a base model. It is not instruction-tuned or safety-tuned.

Results

Training

Zero-shot benchmarks

Scores at the final step, from the run's own evaluation. Shared-9 is the unweighted mean of the nine tasks.

These were measured earlier in the project than the other four checkpoints' scores, so small differences against them (LAMBADA in particular) may come from the evaluation setup as well as the model.

Usage

This is a standard LlamaForCausalLM checkpoint, so it needs no custom code. Tested with transformers==5.8.0.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Mercity/pretrain-baseline-non-qknorm"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

To score text instead of generating it, pass labels and read the loss:

batch = tokenizer("FineWeb is a large web-text dataset.", return_tensors="pt").to(model.device)
with torch.no_grad():
    loss = model(**batch, labels=batch["input_ids"]).loss
print(f"loss={loss.item():.3f}  ppl={loss.exp().item():.1f}")

Model details

Training

Related checkpoints

Limitations

Trained on 6B English web tokens only, a small budget for a 1B model. Training was less stable than the QK-norm rerun, and benchmark scores are single-seed. The model will repeat or make up facts and has had no alignment training.