comp-vs-like-1B-bpe-multi-128k
A 1.2B-parameter Llama-style language model trained with the BPE tokeniser, from Objective vs. Search: Decomposing What Makes a Good Tokeniser (EMNLP 2026).
The paper separates two axes that BPE and UnigramLM confound. This model occupies the cell:
| Compression objective | Log-likelihood objective | |
|---|---|---|
| Bottom-up merging | BPE | BottomUpLL |
| Top-down pruning | TopDownComp | UnigramLM |
This model: BPE — compression objective, bottom-up merging.
All models in the collection are identical except for the tokeniser, so differences in bits-per-byte and BLiMP are attributable to tokeniser design alone.
Training
| Parameters | 1.23B (2048 hidden, 22 layers, 32 heads, 4 KV heads, tied embeddings) |
| Tokeniser | bpe-multi-128k, vocabulary 128,000 |
| Corpus | FineWeb-2 (English + German, Spanish, Turkish, Chinese) |
| Tokens | 20B (76,293 steps x 128 sequences x 2048 tokens) |
| Sequence length | 2048 |
| Optimiser | AdamW (0.9, 0.95), weight decay 0.1, grad clip 1.0 |
| LR | 6e-4, 2000 warmup / 2000 decay steps, min ratio 0.01 |
| Precision | bf16 |
| Seed | 42 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("AhmetcanYvz/comp-vs-like-1B-bpe-multi-128k")
model = AutoModelForCausalLM.from_pretrained("AhmetcanYvz/comp-vs-like-1B-bpe-multi-128k")
ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=20)[0]))
The tokeniser is bundled with the model, so no separate download is needed.
Caveats
This is a research model trained on 20B tokens to study tokenisation. It is not instruction-tuned, not filtered for safety, and not intended for deployment.
Code
https://github.com/Ahmetcanyvz/comp-vs-like
Citation
@inproceedings{yavuz2026objective,
title = {Objective vs. Search: Decomposing What Makes a Good Tokeniser},
author = {Yavuz, Ahmetcan and Meister, Clara and Pimentel, Tiago},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
year = {2026}
}