softmax-209m-16k-nope-1x-s2357
A 209M-parameter research language model from the DilutionAttention release: softmax attention in every layer, no position encoding (control; this seed hit gradient spikes near step 13,400 and never recovered the loss). Trained at a 16,384-token context on 4.19B tokens (1x Chinchilla), seed 2357. It is a base model for studying the attention operator, not an assistant: it is not instruction-tuned or safety-tuned.
Results
| measure | value |
|---|---|
| validation loss (common 1M-token FineWeb-Edu slice) | 3.2907 nats/token |
| NIAH retrieval at 16,000 tokens (in context; chance 1/7) | 0.17 |
| NIAH retrieval at 24,000 tokens (1.5x the trained context) | 0.14 |
| NIAH retrieval at 32,000 tokens (2.0x the trained context) | 0.14 |
| NIAH retrieval at 48,000 tokens (2.9x the trained context) | 0.14 |
| NIAH retrieval at 64,000 tokens (3.9x the trained context) | 0.14 |
| NIAH retrieval at 96,000 tokens (5.9x the trained context) | 0.14 |
| NIAH retrieval at 128,000 tokens (7.8x the trained context) | 0.14 |
| NIAH retrieval at 192,000 tokens (11.7x the trained context) | 0.14 |
| NIAH retrieval at 256,000 tokens (15.6x the trained context) | 0.14 |
| PG-19 loss at positions 8,192-16,384 (in context) | 3.820 nats/token |
| PG-19 loss at positions 16,384-32,768 (2x) | 3.978 nats/token |
| PG-19 loss at positions 32,768-65,536 (4x) | 5.544 nats/token |
| PG-19 loss at positions 65,536-131,072 (8x) | 6.203 nats/token |
NIAH is likelihood-scored keyword retrieval from repetitive filler (see the release's reproducing.md), not long-context reasoning. Every number here, and the matched controls, are in the release's evidence ledger.
Use
pip install "dilution-attention[hub] @ git+https://github.com/Royer-Research-Labs/Dilution-Attention" tiktoken
import tiktoken, torch
from dilution.hub import from_pretrained
model = from_pretrained("Royer-Research-Labs/softmax-209m-16k-nope-1x-s2357", device="cuda") # fused Triton kernels on CUDA
enc = tiktoken.get_encoding("gpt2")
ids = torch.tensor([enc.encode("The history of the printing press")], device="cuda")
with torch.autocast("cuda", dtype=torch.bfloat16):
out = model.generate(ids, 64, temperature=1.0, top_p=0.9, vocab_limit=50257)
print(enc.decode(out[0].tolist()))
Inputs longer than 16,384 tokens need dilution.extend.ExtendedContext(model, length) around the call.
Training
- Architecture: 24 layers, width 768, 12 heads, SwiGLU, no biases, tied embeddings, GPT-2 vocabulary; position encoding: none.
- Data: FineWeb-Edu
sample-100BT(revision fc9850d), a 40B-token GPT-2-tokenized corpus rebuilt withdilution-prepare-data(reproducing.md). - Schedule: 65,536 tokens per step, LR 6e-4 cosine over a 2x-Chinchilla (8.4B-token) schedule; this is the step-64,000 (1x) checkpoint, where the learning rate is 55.7% of peak.
- Run record:
runs/scale_16384/all_softmax_nope_lr6e-4_seed2357/in the release repository (metrics, resolved config, data provenance). Source checkpoint SHA-256:b079ee108edcf5198451383c8bafde44e0806206a1649e620cd8b0e897231c4f, step 64,000.
License and data
Weights: Apache-2.0. Training data: FineWeb-Edu (ODC-By 1.0), used with attribution. The model can produce incorrect, biased or offensive text; it is a research artifact.
Citation
Cite the software release: Royer, N. (2026). DilutionAttention. Zenodo. https://doi.org/10.5281/zenodo.23073037 (BibTeX in the README and CITATION.cff). Technical report: https://doi.org/10.5281/zenodo.23073151.