openeurollm/kda-sigmoid-hybrid-1.3B-100B
Paper: https://arxiv.org/abs/2609.24797
Code: https://github.com/OpenEuroLLM/ComplexKDA
Authors: Julien Siems, Riccardo Grazzi, Korbinian Pöppel, Jaisidh Singh, Arber Zela, Timur Carstensen, Jenia Jitsev, Frank Hutter, Volkan Cevher, Antonio Orvieto, Aaron Klein
A KDA baseline (bounded sigmoid gate) hybrid language model (1.36B parameters) -- linear layers with full attention every 4th layer -- from the ComplexKDA release.
ComplexKDA is Kimi Delta Attention with a signed decay gate: the per-channel
decay alpha is allowed to take either sign, alpha in [-1, 1], instead of
being confined to (0, 1]. That is the one-dimensional real case of a complex
eigenvalue, so a channel can oscillate rather than only forget. The magnitude is
carried in log space exactly as KDA carries it; the +-1 part is carried as a
running product pushed onto the queries and keys, so the recurrence the kernels
run is still the unsigned one.
This checkpoint's decay gate is unsigned sigmoid (the KDA baseline: alpha in (0, 1]).
Architecture
- 24 layers, hidden size 2048, MLP 5312 (SwiGLU)
- 16 heads of dimension 128, short convolution of width 4
- vocabulary 32000, trained at context 4096
- embeddings untied
- attention at layers [3, 7, 11, 15, 19, 23] (gated, NoPE), linear everywhere else
Tokenizer, and how to start a prompt
The bundled tokenizer is configured the way the training corpus was encoded:
no BOS is prepended, documents were terminated with the EOS token, and
model_max_length is this model's trained context. The upstream tokenizer
repository's own defaults differ on both points, so encode through the
tokenizer shipped here rather than re-fetching it by name.
To condition on the start of a document, prefix the EOS token -- that is
what precedes every document's first token in training, and a BOS was never
seen at any position. Leave a continuation bare: mid-document the prefix is a
false signal and costs accuracy. bos_token is remapped to </s> here, so a
caller that asks for "the BOS" gets the separator, while add_bos_token stays
False and the default remains a bare prompt.
Usage
The bundled modeling_complex_kda.py is standalone: torch and
transformers are all it needs.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("openeurollm/kda-sigmoid-hybrid-1.3B-100B")
model = AutoModelForCausalLM.from_pretrained(
"openeurollm/kda-sigmoid-hybrid-1.3B-100B", trust_remote_code=True, dtype="bfloat16")
For the Triton kernels these models were trained with -- much faster, and the exact code path of the training runs -- install the fork:
pip install git+https://github.com/OpenEuroLLM/ComplexKDA
It is picked up automatically when importable. COMPLEX_KDA_BACKEND=torch
forces the portable path; =kernel makes a missing fork an error instead of a
silent fallback.
Provenance
Converted from the training checkpoint with lm_scaling/hf_release/convert_to_hub.py.
The conversion is metadata only -- the weight file is the exporter's own, byte
for byte -- and the bundled implementation is checked against the reference
implementation the runs used.