Neko-V1-Snapshot-1001

A Memory-hierarchical Tiny Language Model

1. Introduction

Neko-V1 is a tiny language model, designed and trained from scratch around a single premise: memory should be hierarchical, not uniform. Its architecture is hybrid and sparse — sliding windows for exact local attention, a recurrent state, a compressed pool, and a hashed lookup — built natively for very long context at a memory cost that does not grow with it. It exists to answer one question: how much of a long context can a tiny model retain without storing a key/value tensor for every token it has read?

This is an early checkpoint. Pretraining ran for only about 4.5 hours on four Blackwell GPUs, and this snapshot is the result of a partial run on roughly 2B tokens. It is not representative of the architecture's final performance; §4 gives the exact figures.

Its answer is to give each part of the stack a different memory resolution, rather than asking one attention mechanism to span the whole range:

• Nekogram — static associative memory. Hashed 2- and 3-grams index independent learned tables at L02 and L10. A content-based gate scores each retrieval against the current hidden state, so a hash collision is something the model can decline rather than something it must live with. No state is carried between steps.
• DeltaNeko — recurrent dynamic memory. Fifteen blocks keep a per-head state matrix updated by a channel-wise decay, erase and write. Lifetimes are initialized log-uniformly across 64 → 128K and shift with depth, so early blocks track local syntax and late blocks hold topic. No positional encoding: position enters only through a short convolution and the recurrence.
• NekoSparse — local exact + global sparse retrieval. Five blocks run sliding-window attention with windows 2K/2K/4K/4K/8K. Three of them also read a 16×-compressed pool of the whole prefix through a Full → Reindex → Reuse chain, so the pool is scanned once, re-selected once, and read once. There is no dense global-attention layer anywhere.
• Nekostreams — depth-wise two-stream residual routing. Two residual streams run in parallel and are mixed at every stage by a matrix that is exactly doubly stochastic by construction, with no normalization step that could fail to converge.

Nothing in the stack allocates a tensor that grows with the sequence.

2. Model Summary

• Architecture: 20 layers — (DeltaNeko × 3 + NekoSparse) × 5; d_model 768, feed-forward 2048 (Smooth-SwiGLU), 2 residual streams
• Parameters: 364M total — 164M neural backbone, 200M lookup tables (tied embedding + Nekogram)
• Context Length: 131,072 tokens
• Vocabulary: 129,280, tied embedding and LM head
• Precision: bfloat16
• License: Apache-2.0

Figure 1. Neko-V1 architecture.

3. Model Architecture

4. Training Progress

This is an early snapshot, not a finished run. The architecture is the result of this release; the weights are a checkpoint along the way.

The base run stopped before its learning-rate decay, so it never received the annealing that carries most of the capability gain, and at about 5.1 tokens per parameter its budget is roughly a quarter of compute-optimal. The instruction tune did complete its schedule, which is why the model follows short instructions rather than merely continuing text.

The consequence is measurable: 24.5% ± 3.4% on MMLU (600 questions, cloze) against a 25.0% chance baseline — no factual recall, and no collapse either. Asked 2+2, it produces a fluent paragraph about algebra and never gives the answer.

5. Usage

pip install "transformers>=4.56" torch safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "AlphaYou/Neko-V1-Snapshot-1001"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
).eval()

messages = [{"role": "user", "content": "Write a Python function that reverses a string."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs, max_new_tokens=256, do_sample=True, temperature=0.6, top_p=0.95,
        eos_token_id=1, pad_token_id=1,
    )
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Apple Silicon, with no torch dependency and reading the same model.safetensors:

cd mlx && pip install -r requirements.txt
python generate.py --prompt "What is 2+2?" --max-new-tokens 64 --temperature 0.6
python chat.py --chat          # interactive; keeps state across turns

Notes:

• trust_remote_code=True is required: this architecture is not built into transformers, so the model code travels with the repository.
• One prompt per forward pass. The model runs a single document per sequence — the recurrent state and sliding windows have no per-position masking — so padding a batch raises an error rather than producing quietly wrong output.
• Sampling at temperature 0.6 / top-p 0.95 is the shipped default. Greedy decoding is noticeably more repetitive. Generation stops at <｜end▁of▁sentence｜> (id 1), which the model does emit.
• Chat template: <｜begin▁of▁sentence｜>[<｜System｜>{system}][<｜User｜>{user}<｜Assistant｜></think>{assistant}<｜end▁of▁sentence｜>]…<｜User｜>{user}<｜Assistant｜></think>. The trailing </think> closes an empty reasoning block and is not optional. There is no thinking mode to enable.
• transformers emits a fix_mistral_regex warning when loading this tokenizer. It is a false positive on a heuristic for a known Llama/Mistral regex; the tokenizer here was checked against the reference encoder on English, Chinese, code and emoji and produces identical ids.

6. Limitations

• Knowledge. Questions that need stored facts get invented answers.
• Reasoning. Multi-step arithmetic and logic are unreliable. There is no chain-of-thought supervision and no scratchpad channel.
• Repetition. Greedy decoding loops; sample, or add a repetition penalty.
• Languages. English is by far the strongest. Chinese is tokenizer-supported and appears in output, but is much weaker.
• Long context. The architecture accepts 131,072 tokens with flat memory, but this checkpoint's reliable context is far shorter. The retrieval path has not been evaluated at 128K.
• Safety. No alignment tuning of any kind. The model will follow harmful instructions and reproduce biases from its training text.
• Batch size 1 only.

Intended for research on hybrid and recurrent attention, memory-bounded long-context inference, and as a starting point for further fine-tuning. Out of scope: production deployment, factual question answering, or anything where an incorrect answer causes harm. There is no moderation layer.

7. License

Weights and inference code are released under Apache-2.0; see LICENSE and NOTICE. This release contains no training code and no training data. The tokenizer is redistributed unmodified from DeepSeek-V4.1-Flash and remains subject to its own terms.

8. Acknowledgements

Neko-V1 is an independent architecture that borrows specific mechanisms from prior work. The correspondences below are implementation-level, not a claim that Neko-V1 is checkpoint-compatible with any of these models.

Also: nanochat (MIT) for the training lineage this architecture descends from; flash-linear-attention (MIT) for the chunked delta-rule kernels used during development; and the Smooth-SwiGLU paper, equation (3), for the channel-scaling formulation.

NVIDIA's GatedDeltaNet-2 reference source carries a non-commercial research licence and is not bundled with this release or used by it; the training-time acceleration this project used is the MIT-licensed FLA implementation.

Citation

@misc{neko_v1_snapshot_1001,
  title  = {Neko-V1-Snapshot-1001},
  author = {AlphaYou},
  year   = {2026},
  note   = {A Memory-hierarchical Tiny Language Model},
  url    = {https://huggingface.co/AlphaYou/Neko-V1-Snapshot-1001}
}