Boris-1.7-D60M-n30M
A 60M-parameter dense decoder-only transformer with a 30M-parameter n-gram embedding table spliced into the residual stream after block 2. One experiment, one variable: does 30M of lookup buy what 30M of dense weight would, at a fixed 2.5B-token budget on a single RTX 3060 12GB?
The trunk is deliberately conventional — the table is the only thing under test.
Model
| Trunk | 10 layers, d=768, 12 heads / 4 KV heads (GQA), SwiGLU 1920, RoPE, RMSNorm, tied embeddings |
| Context / vocab | 1024 / 32,768 (own byte-level BPE, trained on this corpus) |
| Dense (non-embedding) params | 59,981,568 |
| Embedding params | 25,165,824 (tied) |
| N-gram table | 468,751 rows × rank 64 + 64→768 projection = 30,049,216 |
The table is keyed on the literal last 2 and 3 token ids — an explicit top-K
vocabulary (281,250 bigram + 187,500 trigram rows, one shared OOV row), built
by ngram.py over the corpus. No hashing, exact searchsorted lookup, one
gather per position and no matmul, so compute per token barely moves.
Training
- 2.5B tokens: Ultra-FineWeb bulk, then cosmopedia-v2 + fineweb-edu anneal over the final 15%
- WSD schedule (2% warmup, 85% stable, 1-sqrt decay), Muon on 2-D linear maps, AdamW on everything else, bf16 +
torch.compile - 10,172 steps, 245,760 tokens/step, 22h37m wall clock
- Final val: loss 3.235, ppl 25.4, 1.0012 bits/char; n-gram hit rates ~65% bigram / ~21% trigram
Resumable to the byte (optimiser states, data cursor, all RNGs round-trip through the checkpoint).
Results (zero-shot)
Scored with the model's own tokenizer; harness validated against
GPT-Neo-125M's published numbers first. --no-ngram ablates the table at
inference, which measures how much the finished model leans on it — not the
dense control (that needs its own training run).
| Task | metric | intact | table ablated | Δ |
|---|---|---|---|---|
| HellaSwag | acc_norm | 31.22 | 30.54 | −0.68 |
| ARC-Easy | acc_norm | 45.88 | 43.77 | −2.11 |
| ARC-Challenge | acc_norm | 26.02 | 22.95 | −3.07 |
| PIQA | acc_norm | 62.57 | 60.55 | −2.02 |
| WinoGrande | acc | 51.07 | 50.12 | −0.95 |
| LAMBADA | acc | 26.66 | 25.33 | −1.33 |
| val loss | bits/char | 1.0012 | 1.0699 | +0.0687 |
Ablating the table costs 0.069 bits/char and 0.7–3.1 points across the six tasks. The two ARC splits move most; HellaSwag and WinoGrande barely notice.
Additional benchmarks, table intact only:
| Benchmark | metric | score |
|---|---|---|
| ArithMark-3 | acc_norm | 36.1 ±1.5 (random 25) |
| BananaMind Base Bench 1.1 | Elo | 1022 (54.3% acc) |
Usage
This is not a transformers checkpoint — it's a custom architecture (60M
dense trunk + 30M n-gram lookup table) with a custom byte-level BPE tokenizer.
Load it with the loader in this repo:
from loader import load_model, load_tokenizer
model = load_model(".").eval() # reads config.json + model.safetensors
tok = load_tokenizer(".") # reads tokenizer.json (custom 32,768 BPE)
The n-gram table and its top-K lookup arrays are inside model.safetensors,
so that one file — plus config.json — is enough to run the full model.
lm_head.weight is tied to the input embedding and stored once.
Tokenize with the tokenizer's GPT-2-style byte-level split (digit runs are capped at three characters):
ids = tok.encode("The capital of France is")
Generation loop (greedy, single token step):
import torch
def generate(prompt, n_tokens=32):
ids = tok.encode(prompt)
for _ in range(n_tokens):
with torch.no_grad():
logits = model(torch.tensor([ids]))[0]
nxt = int(logits[0, -1].argmax())
ids.append(nxt)
if nxt == 0: # <|endoftext|>
break
return tok.decode(ids)
print(generate("The capital of France is"))
AutoTokenizer also works (self-contained tokenizer.json), but the model
itself has no transformers class:
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("opencerebral/Boris-1.7-D60M-n30M")
Run python loader.py "The capital of France is" for a next-token smoke test.