rwkv-hils-3-m10
A 44.6M-parameter research language model. It interleaves RWKV-7 selective state-space layers (linear-time recurrence; Peng et al., RWKV-7) with HiLS layers — Hierarchical Landmark Sparse attention, which scores chunk summaries with an entropy-calibrated, low-rank-calibrated query, attends the Top-K chunks in full, and always attends a local window (Hu et al., Hierarchical Sparse Attention Done Right). Text-only, byte-level BPE, 4096-token vocabulary, 8192-token position budget.
Deviation from that paper: chunk summaries here use a shared learnable query instead of per-chunk landmark tokens (the paper's "w/o lmk, shared q_c" variant), because landmark tokens need extra tokens interleaved through every layer and that interacts badly with the recurrent blocks in this hybrid.
On the name. m10 is a tier label from this project's own ladder, keyed to
depth and width rather than to parameters — this m10 is 44.6M parameters, not
10M. The ladder exists because the original tier labels were optimistic about
parameter counts; the dimensions below are what is authoritative.
This is a research artifact, not a product. It has no instruction tuning and no safety tuning, and a couple of paragraphs of freeform generation are enough to find its failure modes — they are listed below rather than left for you to discover.
Files
| file | what it is |
|---|---|
model.safetensors | fp32 weights, unmodified state_dict tensor names, 44.6M params; safetensors, so loading executes no code |
config.json | every architecture field plus step, params, model_type |
tokenizer.json | byte-level BPE, 4096 vocab (trained for this project) |
web/manifest.json, web/weights.bin, web/tokenizer.json | fp16 export for the in-browser runtime, half the download |
cases.json | prompts and expected answers used by the browser demo |
chat/ | The chat tuned files |
web/chat | The quantized fp16 weights for the chat version |
Architecture
| parameters | 44,616,292 (fp32) |
| layers | 12 = rwkv7, rwkv7, rwkv7, hils, hils, rwkv7, rwkv7, rwkv7, rwkv7, rwkv7, hils, hils |
| d_model / heads / d_head | 512 / 8 / 64 |
| feed-forward | 4× |
| HiLS chunk / local window / Top-K | 64 / 512 / 16 chunks |
| query-calibration rank | 64 |
hils_share | full — every HiLS layer computes its own K/V and its own Top-K |
| max positions | 8192 |
| vocab | 4096; ids 0–255 are reserved bracket symbols (see below), 256 PAD, 257 SEP, 258+ text |
Two design details are worth knowing before you read the code:
- The vocabulary carries a synthetic task. Ids 0–255 are opening and closing bracket types for a Dyck-language task that shares the sequence with text, which is why a 4096-token text vocabulary sits inside a 4096 total. The model was trained on interleaved bracket and text windows.
- HiLS layers are not attention over the raw sequence. They attend over
chunk summaries and a local window, which is what keeps the cost near-linear
while still allowing exact retrieval inside the window.
q_cal_rankis the low-rank calibration applied to the query before scoring summaries.
Measured behaviour
All numbers below are from the released checkpoint at step 10,000, measured locally (Arc A770, fp32) with the scripts in the repository.
| metric | value | how it was measured |
|---|---|---|
| text loss | 2.307 | 8 windows of 512–1024 tokens, teacher-forced CE, from a differently-seeded slice of the same corpora — see the caveat below |
| Dyck, depth ≥ 10, close-step accuracy | 0.949 | next-symbol accuracy where the target is a closing bracket, which is the position that requires the stack; overall accuracy is ~0.5 by construction, since openings are uniform random |
| needle-in-a-haystack, single needle | 1.00 @ 1K, 1.00 @ 4K | teacher-forced exact match over the answer span, 4 seeds |
| needle-in-a-haystack, multi-needle | 0.00 | same, but this variant was never generated during training |
| needle-in-a-haystack, variable tracking | 0.00 | same, never generated during training |
| needle-in-a-haystack, real-prose families | 0.00 | same; these paraphrase-based families postdate this checkpoint |
The last three rows are the honest headline: retrieval in this model is specific to the retrieval task it was trained on. It solves the single-needle variant perfectly and does nothing at all on variants that never appeared in its data. That is a statement about the training data, not about the architecture — a newer, deliberately broader needle generator exists in the repository and is not what this checkpoint was trained with.
Known failure modes
- Repetition. Long freeform generations collapse into repeating phrases or into short list-like lines. Sampling-side penalties (DRY, a targeted continuation penalty, a frequency penalty) were implemented and measured: they move which attractor it lands in, not whether it lands in one. At this scale it is a property of the model, not of the sampler.
- No instruction following. Base model. Prompts are continuations, not requests.
- Short effective range for prose. Retrieval holds at 4K for the trained variant; general coherence over long documents was not measured here and should not be assumed.
- Incorrect/conflicting chat tokens The chat model was finetuned using tokens that were already present in its output due to an oversight in planning.
- Chat coherence is very weak The chat model is able to sometimes stay on topic, but factuality or general coherence is not effective.
Running it
Python — chat.py at the root of this repository is the entry point
(interactive REPL and one-shot mode):
uv run chat.py Vibelet/rwkv-hils-3-m10 --prompt "The sea was" --max_new 200
uv run chat.py /path/to/local/dir --device cpu # or xpu / cuda
# or install it and use the console script
uv pip install . && rwkv-hils-3 Vibelet/rwkv-hils-3-m10 --prompt "The sea was"
A repository id, a local directory holding model.safetensors + config.json,
a .pt training checkpoint and a directory of checkpoints are all accepted.
Runs on CUDA, Intel XPU and CPU; fp32 inference, ~180 MB resident. The fast
kernels need triton, but a host without it still works: the model falls
back to the reference recurrence, which produces identical tokens (verified
by running with the triton import blocked) and only changes the speed.
Browser — the demo in the repository runs the model entirely client-side, with WebGPU when available and a CPU fallback:
uv run demo/web/serve.py # serves with no-store, port 8123
# open http://127.0.0.1:8123/demo/web/index.html
?backend=cpu forces the reference path. The page can also point at any hosted
copy of the weights with ?model=<base-url>, which is how the public demo site
loads the fp16 export from this repository.
From the Hub directly — download the files and use web/ as the base URL;
the browser runtime fetches one manifest and one flat blob, and views every
tensor in place rather than copying them.
The fp16 export was checked against the fp32 weights in the browser runtime before shipping: max |Δlogit| 9.1e-3 over 24 decode steps, with the greedy token sequence identical. That is a constant quantisation offset and not a backend disagreement, so the demo's CPU-versus-WebGPU comparison still holds within either export.
Training
| steps | 10,000 |
| wall clock | 42 hours |
| precision | fp32 |
| hardware | a single NVIDIA L40(The intial 2000 steps were performed on an Arc A770) |
| learning-rate multiplier | halved once mid-run, automatically, when the needle-in-a-haystack score for the trained variant dropped by more than 0.05 |
| objective | cross-entropy with per-token weights: bracket and text windows at 1.0, needle, question and answer tokens at 5.0 |
Training mixes three kinds of window: synthetic bracket sequences (curriculum over depth), real text, and needle-in-a-haystack examples rendered as text. The mix shifts over the run — bracket-heavy early, text-dominant later, with the needle task present from the point it becomes measurable.
Data. Public corpora, drawn through a cached token pipeline:
| corpus | size in the local cache | provenance |
|---|---|---|
| Wikipedia (enwiki, 20231101 dump) | 238M tokens | Wikimedia, CC BY-SA |
| Project Gutenberg English | 158M tokens | public domain; filtered to Library-of-Congress class P (language and literature) |
| fineweb-edu | 145M tokens | ODC-By, CommonCrawl-derived |
| literature | 8.1M tokens | ACOSharma/literature |
| fiction | 2.2M tokens | public-domain novels, one document per book |
Each listed source was drawn to between 82.5% and 86.7% of its target epochs by final step, so the run consumed these corpora at close to one pass. The pipeline also renders synthetic bracket and retrieval windows. This release ships inference and the browser demo only; the training pipeline is not part of it, and the dimensions, step count, objective and wall clock above are what the released weights were produced with.
Caveat on the text-loss figure. Because the corpora were consumed at close to one epoch, the validation windows — drawn with a different seed from the same sources — are not strictly held out. At 44.6M parameters and under one epoch of exposure, memorisation of any specific window is unlikely, but the number should be read as an in-corpus loss rather than a generalisation measurement. A loss on text the run never saw is not reported here.
Content note: the corpus above is public web text and public-domain literary fiction. The model can produce text that is unsuitable for some audiences and has no filtering of any kind.
Chat conditioning
| steps | 1,000 |
| wall clock | 9 hours |
| precision | fp32 |
| hardware | a single NVIDIA L40 |
| learning-rate multiplier | 2.5e-5 |
| objective | cross-entropy with masked per-token weights |
Data
The single data source for the fine tuning is the wildchat dataset and drew upon 260 million tokens from the dataset. Toxic discussions and non-english language text were filtered out.
Limitations and intended use
Intended for experimentation with hybrid recurrent/attention architectures, sparse-attention retrieval behaviour, and small-model training dynamics. It is not suitable for production, for factual applications, or for any use where outputs need to be correct, safe or attributable. Do not treat its outputs as information about the world.
The chat version is very confident but also almost always very wrong.
Licence
MIT. The training corpora keep their own licences, listed above.