veri-base-8739
1.2B params. Trained from scratch. Not a finetune, not a distill. These weights never existed before this run.
Training logs, benchmarks, architecture explorer: veriepic.dev/model
training
14B tokens, one pass each. FineWeb + DCLM. Cosine schedule that expired halfway (lesson learned: schedules that expire are banned), flat ever since.
The loss curve has a story. It collapsed to ~2.1 and flatlined for thousands of steps while samples stayed word salad — attention was broken from step 1 and the metric was lying. Fixed it, loss jumped back to ~7.5, real descent to 5.13 over 8739 steps. The shaded part is the lie. Everything right of the line is real.
architecture
24 identical blocks, dim 2048. RMSNorm, attention, residual, RMSNorm, SwiGLU, residual. Embeddings tied to the output head. RoPE positions, bf16, 4k context.
remy
Split 16 heads in half, full softmax on each half over the same input, subtract the second from the first with a learned per-pair lambda. Shared junk cancels, signal stays. QK-norm before RoPE, sliding window on even layers, 4 KV heads shared across 16 queries.
Shipped in the 1.19B: the remy family took 4 of the top 5 in the fixed 35-arch zoo, with qk-post edging first by 0.01. The big run committed to remy before the rerun finished — qk-post is data for the next scale, not a recall of this one.
terry
Trained with the private Terry build. Public Terry v2 lives in the main repo and matches it loss-for-loss: factored adaptive tables, grafted matrices, table nesterov, one momentum state per parameter.
| duel | terry | muon | adamw | private |
|---|---|---|---|---|
| 7M · 1000 steps | 2.93 | 3.24 | 3.53 | 3.86 vs 3.84* |
| 140M · H100 · 400 steps | 3.95 | 4.09 | 4.59 | 3.95 |
| states 7M | 28.4MB | 79.7MB | 56.3MB | 4.0MB |
| states 140M | 226.9MB | 380.4MB | 452MB | 61MB |
*matched 300-step run. −9.5% vs muon, −16.9% vs adamw at 7M.
timmy
Custom 65k byte-level BPE, trained in 20 minutes on a rented CPU. Single digits split for arithmetic, think tags atomic.
| test | timmy 64k | o200k | cl100k |
|---|---|---|---|
| plain english | 14 | 16 | 16 |
| veri chat | 23 | 29 | 29 |
| reasoning | 22 | 29 | 29 |
| code | 61 | 56 | 56 |
| math | 38 | 27 | 27 |
| total | 171 | 168 | 170 |
files
| file | size | use |
|---|---|---|
gguf/veri-base-8739-F16.gguf | 2.4GB | GGUF container, custom arch — loads with load_gguf_veri below, not stock runtimes |
transformer/veri.bin | 2.4GB | torch / transformers |
checkpoint/veri-base-8739.pt | 3GB | full bundle to keep training it |
config.json, tokenizer.json | — | the usual |
modeling_veri.py | — | the whole arch in one file, torch only |
modeling_remy.py | — | standalone differential attention |
# gguf container (custom arch — stock llama runtimes only know stock arches)
from modeling_veri import load_gguf_veri
model = load_gguf_veri("gguf/veri-base-8739-F16.gguf", dtype="bfloat16")
# transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("veriepic/Veri-Base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("veriepic/Veri-Base", trust_remote_code=True)
Sampler settings (what the training log uses — greedy and penalty-free sampling babbles, this doesn't): temp 0.7, top-50, frequency penalty 0.2 applied as logits - 0.2 * counts before temperature.
ids = tok("Scientists have discovered that", return_tensors="pt").input_ids.cuda()
with torch.no_grad():
for _ in range(32):
lg = model(ids[:, -4096:]).logits[:, -1, :].float()
counts = torch.bincount(ids[0], minlength=lg.size(-1)).float()
lg = (lg - 0.2 * counts) / 0.7
v, _ = torch.topk(lg, 50)
lg = torch.where(lg < v[:, -1:], float("-inf"), lg)
ids = torch.cat([ids, torch.softmax(lg, dim=-1).multinomial(1)], dim=1)
print(tok.decode(ids[0]))
status
Base weights. Pretraining parked (shards exhausted), SFT in progress. Fluent English, shaky facts — no instruction tuning yet.
Apache 2.0