Escarda-86M-Base
~86M from-scratch decoder. Base sibling of
Quazim0t0/Escarda-86M
(the chat one). Same SpikeWhaleLM stack: MLA, n-gram engram, hash-lookup,
hyper-connections, HRM refine, JEPA / multi-token-prediction. Same ChatML-aware
tokenizer.
JEPA-distilled base. Use it to keep pretraining / SFT. Not a chat assistant.
Related: chat → Quazim0t0/Escarda-86M · demo → Escarda-86M-Chat Space
Trained on Modal credits for the Small Models, Big Adventures Hackathon.
Model summary
| Parameters | ~85.7M (tie_word_embeddings=True) |
| Type | Decoder-only LM (SpikeWhaleLM, model_type: spike_whale) |
| Hidden size / layers | 640 / 16 |
| Attention | 10 heads (head_dim=64), 1 KV head (MQA), MLA low-rank Q/O, decoupled RoPE(16)+NoPE(48), QK-norm |
| Context length | 4096 tokens |
| Vocab | 16,512 (custom length-max tokenizer) |
| License | Apache-2.0 |
Full architecture writeup is on the chat card.
Architecture
SpikeWhaleLM, ~86M, 16 layers, hidden 640, 4096 context, 16,512 vocab, tied embeddings.
- Multi-head Latent Attention (MLA + XSA) - Q and O LoRA-compressed (rank 128); each head splits RoPE dim 16 / NoPE dim 48; 10 query heads, one KV head (MQA); QK-norm.
- Engram n-gram memory - gated table, hashes local n-grams (up to trigrams) into 4,096 rows, mixes back into the residual.
- Hash-lookup layers (×2) - content-addressable features next to the token embeddings.
- Hyper-Connections - learned width-expanded residuals, Sinkhorn routing, instead of a plain add.
- HRM refinement - extra latent pass over hidden states before the output head.
- Multi-Token Prediction (MTP) - DeepSeek-V3-style extra head, more than one next token. Training only.
- FFN is dense. The block can do MoE; MoE is off in this release.
JEPA vs HRM. Escarda trains both HRM refine and JEPA (
use_hrm_refine=True,use_jepa=True). JEPA predicts future latents during training. Byrne drops JEPA and keeps HRM only.
Tokenizer
SpikeTokenizer. Byte-level length-max (greedy longest-match), 16,512 vocab.
Not BPE. Text → UTF-8 → latin-1 bytes → longest vocab key that fits. ChatML-aware.
Atomic specials: <|im_start|>, <|im_end|>, <think>/</think>,
<begin_solution>/<end_solution>, tool-call markers, plus <bos>/<eos>/<pad>/<unk>.
PreTrainedTokenizer in spike_tokenizer.py. Load with
AutoTokenizer.from_pretrained(..., trust_remote_code=True).
Evaluation
Zero-shot multiple-choice, continuation log-likelihood on each task's val/test
splits. byte_ppl is exp(sum_NLL_nats / total_UTF8_bytes) on WikiText-2 test
(tokenizer-independent). BLiMP is fraction of minimal pairs with
logprob(good) > logprob(bad) (12 paradigms × 150). Stderr is binomial
sqrt(p(1-p)/n).
Language modeling
| Metric | Value |
|---|---|
| WikiText-2 byte_ppl ↓ | 2.2228 |
| BLiMP acc ↑ | 0.7144 |
Multiple-choice suite
| Task | acc | ± | acc_norm | ± |
|---|---|---|---|---|
| arc_easy | 0.3801 | 0.0100 | 0.3615 | 0.0099 |
| arc_challenge | 0.1886 | 0.0114 | 0.2235 | 0.0122 |
| hellaswag | 0.2759 | 0.0045 | 0.2832 | 0.0045 |
| winogrande | 0.5162 | 0.0140 | - | - |
| piqa | 0.5843 | 0.0115 | 0.5631 | 0.0116 |
| openbookqa | 0.1300 | 0.0150 | 0.2500 | 0.0194 |
| boolq | 0.5138 | 0.0087 | - | - |
ArithMark-2.0 (AxiomicLabs)
| Metric | Value |
|---|---|
| acc | 0.2536 ± 0.0087 |
| acc_norm | 0.2348 ± 0.0085 |
n = 2,500 · chance = 0.25.
Distilled base. Lowest byte-perplexity in the Escarda family, weaker downstream. PPL is not a capability ranking. Chat behaviour: Escarda-86M. This one if you want a low-loss base to keep training.
Training & token budget
- Tokens: ~20B from-scratch SpikeWhale (~28k steps). This checkpoint is a JEPA-distilled snapshot of that base.
- Token/param: ~233 (20B / 85.7M). About 11-12× Chinchilla's ~20-tokens/param. Over-trained small model on purpose. Inference is the trade.
Fitting Chinchilla's data term to this run's pretrain loss:
L(D) ≈ 2.611 + 77,715 · D^(-0.537) (nats/token, R² = 0.92)
From that:
- Compute-optimal for this 86M ≈ 4.3B → 20B is ~4.6× past compute-optimal.
- Diminishing-returns knee ≈ 22.5B (where +1B buys < 0.005 nats). 20B lands right there.
- Parameter-bound, not data-bound at 20B: capacity term (~0.82 nats) beats the data term (~0.54). Extra tokens do little. Doubling to 40B is projected ~0.07 nats lower loss (~7% PPL) with basically no downstream gain. Next lever is more params, not more tokens. Same reason this distilled base has the lowest PPL in the family without the best downstream scores - it is already on the data-term floor.
Caveats: single-size fit (irreducible loss + capacity floor folded into one constant). Cosine-LR decay inflates the fitted exponent, so treat β as an upper bound. Token counts are anchored to ~20B and scale linearly if that figure is off.
Usage
Custom architecture. trust_remote_code=True. Modeling code is in the repo
(auto_map):
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"Quazim0t0/Escarda-86M-Base", trust_remote_code=True)
Tokenizer is SpikeTokenizer (tokenizer.json, algorithm: length-max).
Load it with spike_tokenizer.py from the project, not AutoTokenizer.
Acknowledgements
Modal credits, Small Models, Big Adventures Hackathon. Released as a base to build on.
Citation
If you use this model, please cite:
@misc{escarda86mbase,
title = {Escarda-86M-Base: A ~86M-parameter SpikeWhaleLM},
author = {Dean Byrne (Quazim0t0)},
year = {2026},
howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Escarda-86M-Base}},
note = {Quazim0t0/Escarda-86M-Base}
}
Escarda vs Byrne - vision family comparison
Byrne = HRM refine. Escarda = Byrne + JEPA on the vision encoder and the LM trunk. Auxiliary only. Zero inference cost.
Vision encoder (DINOv2 teacher-alignment, n=1024 held-out):
| Byrne-VE | Escarda-VE | |
|---|---|---|
| Params | 39.34M | 39.60M (+JEPA head) |
| CLS cosine | 0.776 | 0.771 |
| PATCH cosine | 0.600 | 0.584 |
| JEPA self-consistency | - | 0.040 |
Docling (same held-out doc images, atomic DocTags): both emit well-formed
DocTags. Byrne-Docling is a bit more complete on the hardest samples (closes
</formula>, includes the <code> wrapper), which matches the slightly higher
teacher-alignment. Escarda-Docling is structurally on par and has the JEPA
representation-learning trait.
Pros/cons. Byrne (HRM): higher teacher-alignment, all capacity on distillation fidelity; no self-supervised objective. Escarda (HRM+JEPA): self-supervised neighbour-prediction (richer spatial structure) at zero inference cost, trading ~1-3% teacher-alignment. Same size class.
Family repos: Byrne-VE · Escarda-VE · Byrne-Docling-131M · Escarda-Docling-126M
Update: engram repair (behavior-preserving)
The n-gram Engram in the original weights was degenerate: frozen LSH compressor at init scale hashed every token to bucket 0, so only one table row ever got gradient. This revision rescales the (frozen) compressor and broadcasts the learned bucket-0 vector across all table rows.
Outputs are bit-identical to the previous revision (verified: max logit difference 0.0 across a prompt battery). The only change: the Engram hash now spreads across the full table and every bucket is independently trainable - so if you distill or SFT on top of this base, the n-gram memory will actually learn instead of staying a constant bias.