Tercet
Instruction-tuned chat model (~502M) — Tercet family
-green.svg){kind=icon}
A ~502M hybrid GDN-2 + GQA model, supervised fine-tuned for chat
What this is
Tercet is the supervised fine-tuned (SFT) chat checkpoint for the Tercet family.
- Base (pretrain):
kerzgrr/Tercet-base - Larger successor to
kerzgrr/Couplet - Architecture: hybrid Gated DeltaNet-2 + gated GQA
Training
Pretrain → SFT
| Stage | Details |
|---|---|
| Base | 8.55B-token FineWeb-Edu pretrain (early stop) → Tercet-base |
| SFT mix | HuggingFaceTB/smoltalk + NousResearch/Hermes-3-Dataset + HuggingFaceH4/no_robots + HuggingFaceH4/ultrachat_200k (train_sft / test_sft) + allenai/tulu-3-sft-personas-instruction-following |
| Epochs | 1 full epoch, full deterministic shuffle |
| Assistant targets | 1,190,392,732 |
| Packed tokens | 1,664,747,251 |
| Conversations | 2,240,172 |
| Wall time | 75.4 hours (sum of resumed sessions) |
| Final step | optimizer step 12,702 |
| Weights | EMA (Hub model.safetensors is EMA @ bfloat16) |
| Seq length | 8,192 (packed SFT) |
| Peak LR | 1 × 10⁻⁴ AdamW, cosine → 10% min |
| Final val loss (EMA) | 1.3194 (ppl 3.74) |
Chat template (ChatML)
<|begin_of_text|><|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|>
Generation prompt ends at <|im_start|>assistant\n.
Model Architecture
Same TinyGDN hybrid as the base (501,635,264 parameters):
| Layers | 32 (GDN-2 ×3 + GQA every 4th) |
| Hidden | 1,024 |
| MLP | SwiGLU 2,624 |
| Attention | 8 Q / 2 KV, head dim 128, partial RoPE |
| Linear | Gated DeltaNet-2, 8 heads × 128 |
| Vocab | 49,152 BPE |
IFEval
Official 541-prompt Google IFEval scorer, zero-shot ChatML, greedy decoding (temperature=0, max_new_tokens=1280):
| Metric | Score |
|---|---|
| Prompt-level strict | 19.2% (104 / 541) |
| Instruction-level strict | 32.9% (274 / 834) |
| Prompt-level loose | 21.3% (115 / 541) |
| Instruction-level loose | 35.0% (292 / 834) |
Install & run
pip install torch safetensors tokenizers huggingface_hub
hf download kerzgrr/Tercet inference.py --local-dir .
python inference.py --prompt "What is the capital of France?"
inference.py auto-downloads weights/tokenizer/tiny_gdn/ and auto-installs pinned flash-linear-attention (Windows applies Hub patches). Git is required on PATH.
Interactive chat:
python inference.py
| Flag | Default | Description |
|---|---|---|
--prompt | — | One-shot user message |
--system | — | Optional system prompt |
--temperature | 0.7 | Sampling temperature |
--top-p | 0.9 | Nucleus sampling |
--top-k | 50 | Top-k |
--max-new-tokens | 256 | Max generation length |
--device | cuda if available | cuda / cpu |
Limitations
- Scale: at ~502M parameters this is a research / edge model, not a frontier system
- Dependency: requires
flash-linear-attention; not GGUF / llama.cpp compatible today
Model family
| Model | Stage | Hub |
|---|---|---|
| Monostich | SFT (~100M LLaMA) | kerzgrr/Monostich |
| Monostich-2-base | Pretrain (~150M hybrid) | kerzgrr/Monostich-2-base |
| Monostich-2 | SFT (~150M hybrid) | kerzgrr/Monostich-2 |
| Couplet-base | Pretrain (~268M hybrid) | kerzgrr/Couplet-base |
| Couplet | SFT (~268M hybrid) | kerzgrr/Couplet |
| Tercet-base | Pretrain (~502M hybrid) | kerzgrr/Tercet-base |
| Tercet | SFT (~502M hybrid) | this repo |
Citation
@misc{tercet2026,
title={Tercet: A 502M Hybrid GDN-2 + GQA Chat Model},
author={kerzgrr},
year={2026},
url={https://huggingface.co/kerzgrr/Tercet}
}
Acknowledgments
- flash-linear-attention (Gated DeltaNet-2)
- HuggingFaceTB/smoltalk
- NousResearch/Hermes-3-Dataset
- HuggingFaceH4/no_robots
- HuggingFaceH4/ultrachat_200k
- allenai/tulu-3-sft-personas-instruction-following
- Base:
kerzgrr/Tercet-base
A tercet is a three-line stanza — larger than a couplet, still compact.