TLM-70M
A Tiered Language Model (TLM) pretrained with Tiered Alignment: one set of weights that exposes two behavioral tiers, toggled by a secret permutation key.
- Public tier (C1) — what you get by loading the weights normally. Open to everyone.
- Keyed tier (C2) — reached by applying
key_5pct.json, which permutes 5% of the attention heads and MLP columns. The permutation is self-inverse, so applying it toggles between the two tiers over the same underlying parameters.
TLM-70M is the 60m rung of a scaling ladder. Its non-keyed counterpart
(identical architecture and token budget, trained without a key) is the baseline used for
all public-tier comparisons.
Files
| File | What it is |
|---|---|
model.safetensors | Weights in the C1 (public) configuration |
key_5pct.json | The secret permutation key — swaps over attn_heads and mlp_cols |
config.json, tokenizer.json | GPT-Neo config; GPT-2 BPE tokenizer (vocab 50257) |
Optimizer state is not included; these checkpoints are for inference and evaluation.
Usage
The weights require GPTNeoForCausalLMTiered, not stock GPTNeoForCausalLM:
this architecture defines lm_head with bias=True, and loading through
AutoModelForCausalLM silently drops that trained bias.
from tiered.model import GPTNeoForCausalLMTiered
from tiered.permutation import load_key, apply_permutation, unapply_permutation
model = GPTNeoForCausalLMTiered.from_pretrained("McGill-NLP/TLM-70M")
key = load_key("key_5pct.json") # downloaded from this repo
# C1 (public) — the state the weights ship in
logits_public = model(input_ids).logits
apply_permutation(model, key) # -> C2 (keyed)
logits_keyed = model(input_ids).logits
unapply_permutation(model, key) # back to C1, bit-exactly
Model details
| Total parameters | 67,408,081 |
| Layers / hidden / heads | 16 / 384 / 12 |
| MLP ratio / context | 8 / 2048 |
| Training tokens | 6.74B (100x total params) |
| Steps / global batch | 34,286 / 96 seqs |
| Peak LR / warmup | 5.8e-4 / 1000 steps |
| Data | FineWeb (retain split) |
| Key coverage | 5% |
Architecture follows the DataDecide ladder (arXiv:2504.11393): DataDecide 60M rung (Table 2), verbatim.
A note on the ladder's names
Repository names here are total parameter counts, rounded. The underlying DataDecide rung labels are not parameter counts, so the two orderings differ — the "90M" rung has 117M total parameters, more than the "100M" rung's 99M. Ordered by actual size, the ladder runs:
TLM-20M, TLM-40M, TLM-70M, TLM-100M, TLM-120M, TLM-180M, TLM-230M, TLM-650M
TLM-180M predates this ladder and uses 12 layers rather than 16, so it is not architecturally
part of it; treat it separately when fitting scaling curves.