Tessera-1B-Nano
A language model that predicts its next thought. This is an independent, small-scale
replication of ConceptLM by Paragon Intelligence Labs: an ordinary
HuggingFaceTB/SmolLM2-360M backbone plus a thin causal concept path. Every
4 tokens are pooled into one concept code, a small causal
module guesses the next code, and that guess is fed back into the decoder while it
writes. The surprising part is not that it works - it is what the model turns out
to be reading from the channel.
60-second demo of the comparison Space: the metals rows were captured live from the Space; the photosynthesis row was generated with the Space's own code against this checkpoint on CPU (the ZeroGPU quota ran out mid-capture).
TL;DR
Same billion tokens, same order, same initialization as the unchanged baseline - the only difference is the concept path. Three findings survive every control we threw at them:
- The decoder leans on the channel. Silence the concept feedback and the model measurably stumbles: zeroing it costs +0.1050 nats of held-out NTP loss.
- But it is not reading its own sentence. Feedback predicted from a completely different sentence - another batch element, a Wikipedia passage, even source code - works just as well. The channel carries something, but not sentence identity.
- And it does not (yet) buy token loss. At this budget the concept arm matches the baseline exactly. What the channel carries, why the model wants it, and whether it pays off at scale - that is the open question this study is built to squeeze.
The exact numbers, the preregistered predictions, and the full record: whitepaper at docs/whitepaper.md in the ncp-smol repository.
What the study found
The same held-out batches, scored under four inference modes:
- normal predicted feedback: held-out NTP loss 2.5142.
- feedback silenced: 2.6192 - zeroing the channel costs +0.1050 nats.
- feedback from another sentence in the batch: 2.5150 - indistinguishable from the sequence's own.
The model consults its concept channel at every step, yet what it reads there is interchangeable across sentences, topics, and domains. Whatever the concept path carries, the token path wants it - and a pure token model of the same size, trained the same way, does not miss it. Cornering that signal is the open question; the probes are preregistered in the study repository.
Numbers
The raw record this narrative is built from; per-arm READMEs and the whitepaper hold the full analysis.
| Metric | Value |
|---|---|
| Held-out NTP loss | 2.5142 |
| Held-out perplexity | 12.3568 |
| Zero feedback delta | 0.1050 |
| Shuffled feedback delta | 0.0008 |
| Codebook perplexity / usage | 7.5477 / 85.6% |
| Training tokens | 999,948,288 |
| Tracked compute estimate | $30.27 |
Intervention deltas are increases in held-out NTP loss relative to normal predicted concept feedback, evaluated on identical batches.
Architecture
- chunk size: 4
- product code: 15 segments x 64 entries
- causal concept blocks: 2
- injection point: before token decoder block 2
- NCP target: next continuous concept
- loss:
L_ntp + 1 L_ncp + 1 L_vq
This is a compact ConceptLM-style implementation, not an 8.9B NCP-ArchPreview replica. It omits iterative residual coding, cross-scale residual connections, and the large-scale training recipe.
Training data and provenance
The matched corpus and the training record are pinned in the ncp-smol repository:
packed-cache SHA256 hashes, the complete metric log, trainer state, and the raw
intervention evaluation JSON (also shipped in this repository as eval.json and
metrics.jsonl).
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("yava-code/Tessera-1B-Nano")
model = AutoModelForCausalLM.from_pretrained("yava-code/Tessera-1B-Nano", trust_remote_code=True)
The Tessera family
Three checkpoints, one story, in reading order:
- Tessera-135M-Gate - 135M architecture gate (overfit, published for the scale story)
- Tessera-1B-Nano-Base - 1B-token matched NTP-only control
- Tessera-1B-Nano - 1B-token matched concept arm (this model)
All cards are generated from the run artifacts by the same build_card; the
study repository holds the
whitepaper and full records.
References
- Code and study: https://github.com/yava-code/Tessera-1B-Nano
- ConceptLM: https://arxiv.org/abs/2602.08984
- NCP-ArchPreview: https://arxiv.org/abs/2609.10715