Haiku-base

Pretrained base model for the Haiku family (~655M)

A larger TinyGDN hybrid: Kimi Delta Attention memory plus gated multi-head latent attention

What this is

Haiku-base is the pretrained (base) checkpoint for Haiku, the ~655M successor to the Tercet family.

• Scales kerzgrr/Tercet-base from ~502M to ~655M parameters
• Hybrid Kimi Delta Attention (KDA) recurrent layers + gated MLA (NoPE) full-attention layers
• Own 65,536 BPE tokenizer (not the Tercet 49k vocab)
• This repo is pretrain-only raw text continuation
• Chat model: kerzgrr/Haiku (SFT + APO; fine-tuned from the later v2 pretrain of this architecture)

This base model is for continuation and research. It will not follow instructions reliably.

Model Architecture

Pipeline: Text Prompt → BPE-65K Tokenizer → Haiku Hybrid Decoder (36L) → Next-token Prediction

Hybrid block schedule (×36)

Every 4th layer is gated MLA; the rest are Kimi Delta Attention:

KDA, KDA, KDA, MLA, … (3:1 recurrent-to-attention)

Technical specifications

Training (pretrain)

Install

1) System requirements

• Python 3.10+
• CUDA GPU strongly recommended
• PyTorch with CUDA matching your driver

2) Create an environment

python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate

3) Install PyTorch

Pick the build for your platform from https://pytorch.org. Example:

pip install torch --index-url https://download.pytorch.org/whl/cu124

CPU-only:

pip install torch

4) Install Python deps

pip install safetensors tokenizers huggingface_hub

Flash Linear Attention is installed automatically by inference.py on first run (pinned commit + Windows import patches when needed). Git must be on PATH.

5) Download the inference script

curl -L -o inference.py https://huggingface.co/kerzgrr/Haiku-base/resolve/main/inference.py

# or Hugging Face CLI
hf download kerzgrr/Haiku-base inference.py --local-dir .

The script auto-downloads model.safetensors, config.json, tokenizer.json, and the tiny_gdn/ package from this repo.

Quick start

Single prompt (streams tokens):

python inference.py --prompt "The history of computing begins"

Interactive REPL:

python inference.py

Common options:

Files

kerzgrr/Haiku-base/
  README.md
  inference.py
  requirements.txt
  model.safetensors
  config.json
  tokenizer.json
  tokenizer_config.json
  special_tokens_map.json
  special_token_ids.json
  merges.txt
  vocab.json
  chat_template.jinja
  tiny_gdn/
    __init__.py
    config.py
    model.py
    haiku_layers.py
    nn_common.py

Limitations

• Base model: not instruction-tuned; may ramble or fail at Q&A format
• Scale: ~655M parameters — research / edge prototype, not a frontier model
• Dependency: requires flash-linear-attention (KDA); not GGUF / llama.cpp compatible today
• Context: trained at 2,048; longer windows are experimental

Model family

Citation

@misc{haikubase2026,
  title={Haiku-base: A 655M Hybrid KDA + Gated-MLA Language Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Haiku-base}
}

Acknowledgments

• flash-linear-attention (Kimi Delta Attention)
• FineWeb-Edu
• Tercet family: kerzgrr/Tercet-base
• PyTorch SDPA / Hugging Face Hub + tokenizers

A haiku is three lines — larger than a tercet, still compact.