Tessera-135M-Gate
The smallest member of the Tessera family, and the reason the bigger ones exist.
This 135M architecture-gate checkpoint (a HuggingFaceTB/SmolLM2-135M backbone
with the full Next Concept Prediction path) was trained on a deliberately overfit
TinyStories subset before any large spend, to prove the whole path - concept
pooling, quantized codes, causal concept blocks, feedback - trains end to end.
It is not a language-model quality result, and it is not meant to be: what makes
it worth publishing is what broke, and what the break predicted at scale.
TL;DR
Small and cheap on purpose: this gate run exists to de-risk the 1B-token comparison before spending on it. It earned its keep in both directions.
- All three objectives train (NTP, NCP, VQ fell 41 to 43% over the run), and the decoder causally uses the concept channel even at 135M: zeroing the feedback costs +0.0205 nats of held-out NTP loss.
- It also caught a real failure mode: effective codebook perplexity stayed near 2.4 with usage around a third of the codebook, and shuffled feedback cost nothing - a low-entropy shortcut the 1B-token run later left far behind. Gate small, then scale: some behaviors only appear above a scale threshold.
Numbers
The raw record this narrative is built from; per-arm READMEs and the whitepaper hold the full analysis.
| Metric | Value |
|---|---|
| Held-out NTP loss | 1.8751 |
| Held-out perplexity | 6.5213 |
| Zero feedback delta | 0.0205 |
| Shuffled feedback delta | 0.0000 |
| Codebook perplexity / usage | 2.4087 / 33.0% |
| Training tokens | 16,777,216 |
| Tracked compute estimate | $1.38 |
Intervention deltas are increases in held-out NTP loss relative to normal predicted concept feedback, evaluated on identical batches.
Architecture
- chunk size: 4
- product code: 9 segments x 64 entries
- causal concept blocks: 2
- injection point: before token decoder block 2
- NCP target: next continuous concept
- loss:
L_ntp + 1 L_ncp + 1 L_vq
This is a compact ConceptLM-style implementation, not an 8.9B NCP-ArchPreview replica. It omits iterative residual coding, cross-scale residual connections, and the large-scale training recipe.
Training data and provenance
The matched corpus and the training record are pinned in the ncp-smol repository:
packed-cache SHA256 hashes, the complete metric log, trainer state, and the raw
intervention evaluation JSON (also shipped in this repository as eval.json and
metrics.jsonl).
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("yava-code/Tessera-135M-Gate")
model = AutoModelForCausalLM.from_pretrained("yava-code/Tessera-135M-Gate", trust_remote_code=True)
The Tessera family
Three checkpoints, one story, in reading order:
- Tessera-135M-Gate - 135M architecture gate (overfit, published for the scale story) (this model)
- Tessera-1B-Nano-Base - 1B-token matched NTP-only control
- Tessera-1B-Nano - 1B-token matched concept arm
All cards are generated from the run artifacts by the same build_card; the
study repository holds the
whitepaper and full records.
References
- Code and study: https://github.com/yava-code/Tessera-1B-Nano
- ConceptLM: https://arxiv.org/abs/2602.08984
- NCP-ArchPreview: https://arxiv.org/abs/2609.10715