Qwen2.5-1.5B, un-aliased ChatML turn markers
Qwen/Qwen2.5-1.5B with exactly two embedding rows replaced — the ChatML turn markers
<|im_start|> (151644) and <|im_end|> (151645) — copied from Qwen/Qwen2.5-1.5B-Instruct.
Everything else is bit-identical to the original.
337 of 338 tensors byte-identical to Qwen/Qwen2.5-1.5B
1 tensor differs: model.embed_tokens.weight, rows 151644 and 151645
Tokenizer, vocabulary, token ids, config and architecture are unchanged. The -Instruct rows are
used because it shares this tokenizer and vocabulary, so the ids line up exactly.
Why
The base checkpoint ships 267 embedding rows that are bit-identical to one another, and both ChatML turn markers are inside that block:
| token | id | rows bit-identical to it |
|---|---|---|
<|im_start|> | 151644 | 97 |
<|im_end|> | 151645 | 267 |
<|endoftext|> | 151643 | 1 (unique) |
tie_word_embeddings: true, so those rows are also the output head. Identical rows produce
identical logits for any residual stream. A model therefore cannot prefer <|im_end|> over its
266 twins, and if the embedding is frozen — as under LoRA, or any adapter method with
modules_to_save: null — no amount of training can separate them.
The failure is quiet and easy to misread as undertraining. On a model fine-tuned against the
unpatched base, we measured 10.7% of probability mass on the aliased block at the turn
boundary — the turn-end intent was learned — but split 267 ways it gave
p(<|im_end|>) = 0.00040, losing the argmax to an untrained token. Generations never terminated.
Note that <|endoftext|>, the tokenizer's declared eos_token, is unique and was never the
problem; a check that only inspects eos_token will miss this entirely.
Effect
Qwen/Qwen2.5-1.5B | this model | |
|---|---|---|
rows bit-identical to <|im_start|> | 97 | 1 |
rows bit-identical to <|im_end|> | 267 | 1 |
p(<|im_end|>) at the turn boundary¹ | 0.00040 (rank 6–9) | 0.98702 (rank 1) |
| generations terminating before the token cap¹ | 0 / 12 | 200 / 200 |
¹ measured with a fixed LoRA fine-tune held constant across the two bases.
Caveats
- These rows are still low-norm — about 0.41× the median embedding row. The fix makes them addressable, not strong. A model that needs to emit them still has to learn to.
- Only the two ChatML markers were separated. The checkpoint still contains 2,210 rows in
81 mutually-aliased groups (largest: 473 rows; the remaining 266 of
<|im_end|>'s original block is one of them). That is harmless for ordinary use — those are unused vocabulary slots — but any token you add or repurpose from them inherits the same problem. Run the check above on every special token your chat template emits. - This is a patched base, not an instruction-tuned model. It has no chat behaviour of its own.
Checking a base model for this
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "interpretable-finetuning/qwen2.5-1.5b-unaliased"
tok, m = AutoTokenizer.from_pretrained(mid), AutoModelForCausalLM.from_pretrained(mid)
E = m.get_input_embeddings().weight.data
for t in ("<|im_start|>", "<|im_end|>"):
i = tok.convert_tokens_to_ids(t)
print(t, "aliased with", int((E == E[i]).all(-1).sum()), "rows") # must be 1
Any base whose chat-template special tokens report more than 1 will not learn to emit them.
Used by
interpretable-finetuning/topklora-qwen2.5-1.5b-v2
— 46 TopK-LoRA sleeper-agent organisms trained on this base. Those adapters are only correct on
this model; loading them on vanilla Qwen/Qwen2.5-1.5B silently restores the broken behaviour.