Tercet-base

Pretrained base model for the Tercet family (~502M)

A larger TinyGDN hybrid language model: recurrent GDN-2 memory plus gated GQA

What this is

Tercet-base is the pretrained (base) checkpoint for Tercet, the ~502M successor to the Couplet family.

• Scales kerzgrr/Couplet-base from ~268M to ~502M parameters
• Hybrid Gated DeltaNet-2 recurrent layers + gated GQA full-attention layers
• This repo is pretrain-only raw text continuation
• Instruction-tuned / chat (SFT): kerzgrr/Tercet

If you want chat behaviour, use kerzgrr/Tercet — this base model is for continuation / research and will not follow instructions reliably.

Model Architecture

Pipeline: Text Prompt → BPE-49K Tokenizer → TinyGDN Hybrid Decoder (32L) → Next-token Prediction

Hybrid block schedule (×32)

Every 4th layer is full attention; the rest are Gated DeltaNet-2:

GDN2, GDN2, GDN2, GQA, … (3:1 recurrent-to-attention)

Technical specifications

Training (pretrain)

Stopped early when EMA validation loss flattened (~3.004), rather than completing a full packed 10B-token epoch.

Install

1) System requirements

• Python 3.10+
• CUDA GPU strongly recommended
• PyTorch with CUDA matching your driver

2) Create an environment

python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate

3) Install PyTorch

Pick the build for your platform from https://pytorch.org. Example:

pip install torch --index-url https://download.pytorch.org/whl/cu124

CPU-only:

pip install torch

4) Install Python deps

pip install safetensors tokenizers huggingface_hub

Flash Linear Attention is installed automatically by inference.py on first run (pinned commit + Windows import patches when needed). Git must be on PATH.

5) Download the inference script

curl -L -o inference.py https://huggingface.co/kerzgrr/Tercet-base/resolve/main/inference.py

# or Hugging Face CLI
hf download kerzgrr/Tercet-base inference.py --local-dir .

The script auto-downloads model.safetensors, config.json, tokenizer.json, and the tiny_gdn/ package from this repo.

Quick start

Single prompt (streams tokens):

python inference.py --prompt "The history of computing begins"

Interactive REPL:

python inference.py

Common options:

Pretrain vs chat

Files

kerzgrr/Tercet-base/
  README.md
  inference.py
  requirements.txt
  model.safetensors
  config.json
  tokenizer.json
  tokenizer_config.json
  special_tokens_map.json
  special_token_ids.json
  merges.txt
  vocab.json
  tiny_gdn/
    __init__.py
    config.py
    model.py

Limitations

• Base model: not instruction-tuned; may ramble or fail at Q&A format
• Scale: ~502M parameters — research / edge prototype, not a frontier model
• Dependency: requires flash-linear-attention; not GGUF / llama.cpp compatible today
• Context: trained at 2,048; longer windows are experimental
• Early stop: pretrain did not finish a full 10B-token epoch

Model family

Citation

@misc{tercetbase2026,
  title={Tercet-base: A 502M Hybrid GDN-2 + GQA Language Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Tercet-base}
}

Acknowledgments

• flash-linear-attention (Gated DeltaNet-2)
• FineWeb-Edu
• Couplet family: kerzgrr/Couplet
• PyTorch SDPA / Hugging Face Hub + tokenizers

A tercet is a three-line stanza — larger than a couplet, still compact.