LemonSeed 1.5B — Base

A from-scratch 1.5B-parameter hybrid Mixture-of-Experts language model, pre-trained end-to-end on a single AMD MI300X (ROCm) using an MLX training stack. This is a base model — pre-training only, no instruction/chat fine-tuning yet.

Author: Geramy L. Loveless · Trained Aug 2026 · Framework: MLX (ROCm backend)

▶ Primary engine: LSE — Lemon Seed Engine (C++ · HRX/ROCm) · 📖 Docs: lemonseed-docs

TL;DR

Architecture

LemonSeed is a hybrid — it interleaves linear-attention, softmax-attention, and sparse-MoE feed-forward, with a Mixture-of-Depths gate on top:

• Sequence mixing (per layer):
  • Gated DeltaNet (linear attention) on most layers — O(N) compute, O(1) recurrent state → cheap long context.
  • Full softmax attention every 4th layer (global attention layers 3, 7, 11, 15, 19), sliding window 256, RoPE θ=500000.
• Feed-forward: Mixture-of-Experts — 8 routed experts, top-2 routing, 1 always-on shared expert, expert intermediate 2176, DeepSeek aux-loss-free load-balancing bias, score band 0.15 (see note).
• Mixture-of-Depths: each layer processes only ~top-15% of tokens through the expert path (mod_top_k=3, threshold 0.15).
• Shape: 20 layers · hidden 1024 · 16 attn q-heads / 2 kv-heads (GQA) · 8 GDN qk-heads / 4 v-heads · RMSNorm.

Full config is in config.json.

Prefill & decode — the algorithm

LemonSeed's efficiency comes from the hybrid: most layers are linear-attention (Gated-DeltaNet) with an O(1) recurrent state, and only every 4th layer is softmax attention with a growing KV cache. So decode is cheap and mostly constant-memory.

Per-layer state carried between steps:

• GDN layers → a fixed-size recurrent state S (a [v_heads, head_dim, head_dim] matrix). Does not grow with sequence length.
• Attention layers (3, 7, 11, 15, 19) → a standard KV cache (grows with context; sliding-window 256).

Prefill — process a prompt of length T (one pass)

h = embed(tokens)                         # [T, D]
for each layer L:
    x = rmsnorm(h)
    if L is attention:
        q,k,v = proj(x)                   # GQA: 16 q-heads, 2 kv-heads
        y = causal_softmax_attn(q,k,v)    # RoPE, sliding window 256
        cache[L].kv = (k, v)              # save for decode
    else:  # Gated DeltaNet (linear attention)
        # chunked delta-rule scan over the T tokens:
        #   S_t = S_{t-1} (I - beta_t k_t k_tᵀ) + beta_t v_t k_tᵀ   (delta rule)
        #   y_t = q_t · S_t
        y, S = gdn_scan(x)                # O(T); keep final S
        cache[L].state = S               # save for decode
    h = h + out_proj(y)                  # residual

    x = rmsnorm(h)
    keep = mod_gate(x)                    # Mixture-of-Depths: ~top-15% of tokens
    r = moe(x[keep])                      # routed: top-2 of 8 experts (+ shared always-on)
    h = h + shared(x) ; h[keep] += r      # scatter routed output back
logits = tie_embed(rmsnorm(h))[-1]        # distribution for the FIRST new token

Decode — generate token t+1 from the saved state (one token in, one out)

h = embed(new_token)                      # [1, D]
for each layer L:
    x = rmsnorm(h)
    if L is attention:
        q,k,v = proj(x)
        cache[L].kv.append(k, v)          # KV grows by 1 (window-capped at 256)
        y = attend(q, cache[L].kv)        # O(context) — but only ~5 layers
    else:  # GDN — O(1), constant memory
        S = cache[L].state
        S = S*(I - beta·kkᵀ) + beta·v·kᵀ  # in-place delta-rule update
        y = q · S                         # read
        cache[L].state = S
    h = h + out_proj(y)

    x = rmsnorm(h)
    if mod_gate(x):                       # is this token routed this layer?
        h = h + shared(x) + moe_top2(x)
    else:
        h = h + shared(x)
logits = tie_embed(rmsnorm(h))            # sample -> next token, repeat

Why it's fast: 15 of 20 layers are GDN → O(1) per decode step, no KV growth. Only the 5 attention layers pay O(context). And MoE runs just top-2 of 8 experts per token, so ~0.46 B active params/token despite 1.5 B nominal. Net: a small KV footprint and near-constant-time decode.

Training

• From scratch (random init) — no distillation, no continued-pretrain from another model.
• Objective: next-token cross-entropy + load-balance aux loss + router z-loss.
• Optimizer: Fused AdamW (bf16 params + fp32 master), cosine LR (peak 3e-4 → ~3e-5), weight decay 0.1.
• Batch: 4 × 2048 × grad-accum 2 = 16,384 tokens/step.
• Corpus (v7, ~9.6 B unique tokens, ~3 epochs):

  domain	share
  code	45.1%
  math	15.2%
  web	10.7%
  logic	10.4%
  docs	9.7%
  conversational	7.6%
  agent	1.3%

• Curriculum: domain-ordered (per-block loss swings 2.5–3.7 are the curriculum, not instability).

Honest capacity note ⚠️

By weight, only 68 of 160 routed experts actually trained (the rest sit near zero-init), so the effective size is ~0.90 B, not the full 1.51 B. The router dispatched tokens across all experts (balanced), but the tight 0.15 score band ran the model effectively top-1 for most of training, starving non-top experts of gradient — a classic winner-take-all collapse, worst in the early layers, full in the deep layers.

This is lost parameter budget, not lost quality — the model works well on what it trained, and MoE only runs the top-2 per token, so dead experts cost memory but no compute. It's documented transparently here rather than hidden. The next model uses a wider score band from step 0 and a per-layer expert schedule to fix it.

Using the untrained experts — expansion slots for post-training (RL / tools / agents)

Here's the useful part: those 92 untrained experts (~0.6 B of near-zero capacity) are empty expansion slots. You can deliberately fill them during post-training with new skills — tool-calling, agentic planning, chain-of-thought reasoning — without growing the model and with catastrophic-forgetting protection built in (freeze the trained base, train only the fresh experts). The inference cost stays the same (still top-2 per token).

Why this works in post-training when mid-pretrain revival did not: a converged base has no unmet demand, so any expert you re-seed is redundant and gets pruned within ~1k steps (we tried; it collapses). But a new task the model can't yet do manufactures demand — the task gradient has nowhere good to go in the existing experts, so the fresh experts find a real niche and stick. You're not reviving generic capacity; you're colonizing empty slots with a skill that needs them.

Which experts are dead (target map, from the final checkpoint)

Note the dead capacity sits in the early/mid layers; the deep layers (14–19) are full. So mid-level skills (structural patterns, composition, formatting like tool-call syntax) map cleanly onto the available slots; the deep layers will consume the new mid-layer features downstream.

How to target them (recipe)

1. Re-seed the dead experts to a usable scale — clone a live sibling in the same layer (or the shared expert for fully-dead layers) + ~10% noise, so they start with useful features (random init just gets pruned). Also clone the router row so they get competitive routing weight.

   # in the lemonseed repo — targeted, incremental, or whole-model:
   python scripts/revive_experts.py IN.safetensors OUT.safetensors \
       --clone --noise=0.1 --layers=2,5,6,7,8,11 --max-per-layer=2
   

2. Widen the score band so non-top experts actually receive combine-weight + gradient (the tight 0.15 band is what starved them):

   export LEMONSEED_MOE_SCORE_BAND=0.5     # true top-2
   

3. Freeze the base, train only the fresh experts — this forces the new skill into the empty slots and prevents forgetting (the trained backbone + live experts can't drift). Unfreeze everything for a short final anneal if you want.
4. Drive it with task demand — SFT on tool-call / agentic / reasoning traces, then RL (GRPO/PPO). The strong, specific reward signal is what makes the re-seeded experts stick and specialize.
5. Monitor with scripts/expert_health.py each ~1k steps — a colonized expert's w_up std should climb (staying above 1e-4) instead of decaying back. If it decays, the layer had no demand for it; move to a different layer.

Start with the sparse layers (2, 5, 6, 7, 8, 11) — they have both a live sibling to clone from and open slots, so they take most reliably. The fully-dead layers (0, 1, 9, 10, 13) are the stretch goal (seed from the shared expert) and need the strongest task signal.

Tooling for all of this — revive_experts.py (re-seed/clone), expert_health.py (measure), and the LEMONSEED_MOE_SCORE_BAND knob — ships in the lemonseed-docs repo.

Capabilities & intended use

• Base model for research / continued fine-tuning. It is not instruction-tuned — it completes text, it doesn't follow chat instructions out of the box.
• Arithmetic: solid with chain-of-thought / scratchpad prompting (the Qwen3.6 digit-split tokenizer helps); one-shot answers are noisier.
• Good starting point for: SFT, RLHF/GRPO, long-context extension, and MoE research.

How to run

LemonSeed uses a custom architecture (hybrid GDN + MoE + MoD), so it is not loadable via transformers. Files here: model.safetensors, config.json, tokenizer.json.

▶ Primary engine — LSE (Lemon Seed Engine)

The reference runtime is LSE — Lemon Seed Engine: a modular C++ training and inference engine built for this architecture, running on the HRX native ROCm runtime. Ops record into a lazy DAG, fuse, JIT to AMDGPU code objects via amd_comgr, and dispatch through the HRX ABI (not HIP). It's the performant, first-class way to load and serve LemonSeed.

git clone https://github.com/Geramy/LSE && cd LSE
cmake -S . -B build -GNinja -DCMAKE_BUILD_TYPE=RelWithDebInfo && cmake --build build
# point it at model.safetensors + config.json + tokenizer.json from this repo

Reference path (Python / MLX)

For a simple/reference implementation, the MLX loader:

from lemonseed.model import load_model
from lemonseed.tokenizer import load_tokenizer
from lemonseed.generate import generate_text

model = load_model("model.safetensors")          # + config.json alongside
tok   = load_tokenizer("qwen3.6")                # or point at the bundled tokenizer.json
print(generate_text(model, tok,
    "Question: What is 8 × 6?\nLet's think step by step.\nAnswer:",
    max_new_tokens=64, greedy=True))

See example_generate.py for a template, and github.com/Geramy/lemonseed-docs for the full architecture + prefill/decode reference.

License & attribution

• Model weights: released under a research / non-commercial license (license_name: lemonseed-research) — see the caveat below.
• Tokenizer: the bundled tokenizer.json is the Qwen 3.6 tokenizer, © Alibaba Cloud / Qwen, under its own license.
• Training data attribution: the corpus includes openly-licensed educational material, notably MIT OpenCourseWare (CC BY-NC-SA). Because that source is NonCommercial, this model is intended for research / non-commercial use. If you plan commercial use, review the data licensing first. Credit to MIT and the open-education community — solo foundation-model work runs on open resources.

Limitations

• Small (~0.9 B effective) base model — expect hallucination, limited world knowledge, no safety tuning.
• Not instruction-tuned; no chat template applied.
• ~half the routed experts are untrained (see capacity note) — a fine-tune (esp. one that colonizes the dead experts) can recover capacity.

Built entirely on MLX (ROCm backend) — hat tip to the MLX maintainers. Trained through a machine migration and a full-disk crash, finished clean at val loss 1.78.