Meridian-Tiny

A ~12M-parameter language model trained from scratch on a single RTX 3060, built to test whether a stack of recent architecture ideas can be combined, scaled down, and trained on consumer hardware.

Meridian-Tiny is a research toy, not a useful assistant. It produces locally plausible, globally nonsensical text. What makes it interesting is what's inside: hybrid linear/full attention, a latent mixture-of-experts, manifold-constrained hyper-connections, attention residuals across depth, and hashed n-gram memory, all in one model small enough to train in minutes.

Everything here (tokenizer, data pipeline, model code, training loop) was written from scratch and verified component by component against reference implementations.

Architecture

Each layer is built from up to three sublayers, each wrapped in its own hyper-connection:

1. Engram (selected layers only): hashed n-gram memory
2. Mixer: KDA linear attention (3 of every 4 layers) or MLA full attention (every 4th layer)
3. FFN: dense SwiGLU in the first layer, latent MoE everywhere else

Implementation notes

This is a from-scratch interpretation, not a reproduction of any official code. Deviations worth knowing:

• mHC + AttnRes together is this project's own integration choice. mHC handles the residual within each block; AttnRes rebuilds the 4-stream state at block boundaries. No published model combines them, as far as I know.
• The KDA forget gate is parameterized as log α = lower_bound · sigmoid(W x) with lower_bound = −5. The fast path uses the chunk_kda kernel from flash-linear-attention, verified to match a hand-written recurrent reference (max abs diff ~2e-4).
• Engram skips the original vocabulary-compression step, uses one shared table per Engram layer with distinct hash multipliers per head, and uses a simplified context gate.
• The MoE runs all experts in batched matmuls using fixed-capacity buffers (capacity factor 2.0). Overflow assignments are dropped for that expert. Verified to match a per-expert loop reference when no tokens overflow.

Model details

Most of Meridian-Tiny's parameters are the embedding and the Engram table. The part of the network doing actual computation is around a million parameters per token.

Tokenizer

Byte-level BPE with a 32,768-token vocabulary, trained on ~1.05B characters sampled from the training mix (Japanese upsampled for the tokenizer only).

• Byte fallback, so no input is ever un-tokenizable
• Japanese script splitting: pre-tokenization splits at kanji / hiragana / katakana boundaries (with ー treated as katakana), so merges follow word stems and inflections
• Individual digits with the leading space attached ( 12 → 1, 2), for more consistent arithmetic
• No Unicode normalization (NFKC would flatten math symbols like x²)
• Special tokens: <|pad|> = 0, <|bos|> = 1, <|eos|> = 2, plus 13 reserved slots

Measured efficiency (characters per token): English 3.38, German 3.07, Russian 2.20, Japanese 1.67, Python 2.38, Rust 2.57.

Training data

A 4B-token corpus was packed; Meridian-Tiny saw 20M tokens of it, sampled according to this mix:

Documents are separated by <|eos|>. Training samples are random 513-token windows drawn from per-source token files, so the mix can be changed without re-tokenizing. No additional quality filtering was applied beyond what the source datasets provide.

FineWeb-Edu, FineMath and FineWeb-2 are released under ODC-By 1.0. StarCoderData is derived from The Stack; code in it remains under its original licenses and is subject to the dataset's terms of use.

Training

Results

Validation loss (nats per token) on held-out data from each source, at the end of training:

Code and math reached lower loss than English prose, since they're highly structured. The non-English languages lag, as expected given their small share of the data, but each improved substantially (Japanese: 10.85 → 7.00).

Ablations

Each component was removed one at a time and trained under identical conditions: same hyperparameters, same 20M tokens, same data order. The seed1 row is the unmodified baseline with different initial weights, which measures run-to-run noise.

Noise level. The two baseline seeds land 0.045 apart on the mix loss, so differences smaller than roughly 0.05 can't be distinguished from luck. Per-source numbers are noisier still (math alone differs by 0.15 between seeds), so the mix column is the most reliable signal.

Engram clearly helps. Removing it costs 0.184 nats on the mix, about 4× the seed noise, making it the only component with a measurable effect at this scale. The cost is not evenly spread. Compared with the average of the two baseline seeds:

This matches what Engram is for: a lookup memory for short repeated phrases. Code is dense with exact recurring n-grams (self., std::, let mut), so it benefits most. The non-English languages, at 2–4% of the data each, likely haven't seen their n-grams often enough for the table to become useful yet.

AttnRes and mHC show no measurable effect here. Removing either changes the mix loss by about 0.04, within seed noise. Both mechanisms route information across depth, and a 4-layer model offers very little depth to route across. This is a "not demonstrated at this scale" result, not evidence that they're useless.

mHC has a real speed cost. Carrying 4 residual streams, plus a Sinkhorn projection for every sublayer, made the baseline noticeably slower than the 1-stream variant, which trained at roughly 38–60K tokens/s. At this scale, dropping mHC gives the same loss for less compute. Whether the extra streams pay for themselves at greater depth is an open question; the full-size model keeps them to find out.

Samples

Temperature 0.8, top-k 50. Prompts in bold.

fn main() { let mut result = 0; ...then collapses into a repetition loop.

def fibonacci(n): produces correctly indented Python with return, elif, except and np. calls, with nonsensical semantics.

猫は その中で、私は、その後には、その方は「その方」を出し、自分の関して、そして、と言わたと思って…

Die Katze sitzt, was unsightly developed in many areas...

The last one shows an interesting failure: German drifts into English at "was" (a word in both languages), while Russian and Japanese prompts stay in their own language. Scripts that don't share tokens with English seem to act as a barrier against drifting into the dominant language.

What's next

Meridian-Tiny was the smoke test. A full-size configuration is training now on the same code and data, keeping every component, including mHC, to see whether depth changes the ablation picture:

Usage

Meridian-Tiny uses custom model code. Download model.py, presets.py, tokenizer.json and model.safetensors from this repo, then:

import torch
from safetensors.torch import load_file
from tokenizers import Tokenizer
from model import MiniMeridian
from presets import TINY

tok = Tokenizer.from_file("tokenizer.json")
net = MiniMeridian(TINY["model"])
net.load_state_dict(load_file("model.safetensors"))
net.eval()

ids = torch.tensor([tok.encode("fn main() {").ids])
with torch.no_grad():
    for _ in range(40):
        logits = net(ids)[:, -1, :] / 0.8
        next_id = torch.multinomial(torch.softmax(logits, dim=-1), 1)
        ids = torch.cat([ids, next_id], dim=1)
print(tok.decode(ids[0].tolist()))

Dependencies: torch, tokenizers, safetensors. Optional: flash-linear-attention for the fast KDA kernel on GPU (plus triton-windows on Windows). Without it, the model falls back to a pure-PyTorch recurrent implementation that's slower but numerically equivalent.

Generation recomputes the full sequence for each token (no KV cache), which is fine at this size.

Limitations

• Not useful for any real task. ~1M active compute parameters and 20M training tokens produce fluent-looking fragments, not meaning.
• Not instruction-tuned, not safety-tuned. It continues text; it does not follow instructions.
• Prone to repetition loops, especially in code and math.
• German output tends to drift into English.
• Trained on unfiltered-beyond-source web text, and may reproduce anything found in it.

Acknowledgements

Architecture ideas from Moonshot AI (Kimi Linear, KDA, attention residuals), DeepSeek (MLA, MoE routing, mHC, Engram) and Qwen (gated attention). Fast KDA kernel from the flash-linear-attention project. Data from Hugging Face (FineWeb-Edu, FineMath, FineWeb-2) and BigCode (StarCoderData).