Byrne-86M
Main Byrne-86M chat checkpoint (OPD v2). The one I point people at for general
use. ~86M SpikeWhaleLM from scratch - MLA, n-gram engram, hash-lookup,
hyper-connections, HRM refine, MTP. Custom ChatML-aware tokenizer.
Trained with Modal credits during the Small Models, Big Adventures Hackathon.
Related: base → Byrne-86M-Base
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-86M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-86M", trust_remote_code=True)
Architecture
SpikeWhaleLM, ~86M, 16 layers, hidden 640, 4096 context, 16,512 vocab, tied embeddings.
- Multi-head Latent Attention (MLA + XSA) - Q and O LoRA-compressed (rank 128); each head splits RoPE dim 16 / NoPE dim 48; 10 query heads, one KV head (MQA); QK-norm.
- Engram n-gram memory - gated table, hashes local n-grams (up to trigrams) into 4,096 rows, mixes back into the residual.
- Hash-lookup layers (×2) - content-addressable features next to the token embeddings.
- Hyper-Connections - learned width-expanded residuals, Sinkhorn routing, instead of a plain add.
- HRM refinement - extra latent pass over hidden states before the output head.
- Multi-Token Prediction (MTP) - DeepSeek-V3-style extra head, more than one next token. Training only.
- FFN is dense. The block can do MoE; MoE is off in this release.
JEPA vs HRM. Byrne is Non-JEPA: HRM refine only (
use_hrm_refine=True,use_jepa=False). Escarda adds JEPA on top of HRM.
Tokenizer
SpikeTokenizer. Byte-level length-max (greedy longest-match), 16,512 vocab.
Not BPE. Text → UTF-8 → latin-1 bytes → longest vocab key that fits. ChatML-aware.
Atomic specials: <|im_start|>, <|im_end|>, <think>/</think>,
<begin_solution>/<end_solution>, tool-call markers, plus <bos>/<eos>/<pad>/<unk>.
PreTrainedTokenizer in spike_tokenizer.py. Load with
AutoTokenizer.from_pretrained(..., trust_remote_code=True).
Evaluation
Zero-shot multiple-choice, continuation log-likelihood (acc_norm =
byte-length-normalized).
| Task | acc | acc_norm |
|---|---|---|
| arc_easy | 0.3670 | 0.3468 |
| arc_challenge | 0.1894 | 0.2355 |
| hellaswag | 0.2815 | 0.2858 |
| winogrande | 0.5201 | - |
| piqa | 0.5756 | 0.5593 |
| openbookqa | 0.1460 | 0.2440 |
| boolq | 0.3865 | - |
ArithMark-2.0 (AxiomicLabs)
- official metric is raw
acc: 0.3096.
Language modeling: WikiText-2 byte_ppl (↓) 2.6839 · BLiMP (↑) 0.7033.
Citation
If you use this model, please cite:
@misc{byrne86m,
title = {Byrne-86M: A ~86M-parameter SpikeWhaleLM},
author = {Dean Byrne (Quazim0t0)},
year = {2026},
howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-86M}},
note = {Quazim0t0/Byrne-86M}
}
Update: format-blended SFT on the engram-repaired base
This revision applies the (behavior-preserving) engram repair, then a short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk, GSM8K-train (with '#### N' reasoning), and MMLU-style ('Answer: ') examples -- so chat fluency improves while the benchmark output-formats are preserved rather than overwritten. Held-out (test-split) before->after:
MMLU acc 0.260->0.254, format 0.918->0.951; GSM8K '####' 0.420->0.815
Note: these are fluency + output-format gains. Benchmark accuracy remains near the floor for a model this size -- the SFT does not add reasoning ability.