Kiseki-TTS-1.1

A small, fast Japanese TTS model with a Mamba2 state-space decoder, built on top of Qwen/Qwen3-TTS-Tokenizer-12Hz.

Kiseki-TTS generates discrete neural audio codec tokens at 12.5 Hz and decodes them to waveform with the Qwen3 TTS codec. Because the acoustic decoder is a linear-time SSM rather than a self-attention stack, generation cost is constant per frame — memory does not grow with utterance length, and there is no KV cache to manage.

Example

Model Details

Model Description

• Developed by: telecomadm1145
• Model type: Encoder–decoder (Transformer encoder + cross-attention/Mamba2 decoder)
• Language: Japanese (ja)
• License: MIT
• Finetuned from: telecomadm1145/Kiseki-1.1-0.3B (a seq2seq translation model)
• Audio codec: Qwen/Qwen3-TTS-Tokenizer-12Hz

Architecture

The decoder deliberately has no causal self-attention. Temporal context is carried entirely by the Mamba2 recurrent state; text conditioning enters through cross-attention whose K/V are computed once from the encoder and reused for every frame.

Audio tokenization

Depth modelling (MTP head). Each frame's 16 codebook layers are predicted by a shared "multi-token prediction" head rather than 16 separate decoder passes. Layer q sees the decoder hidden state plus the exclusive prefix sum of the embeddings of layers 0 … q-1:

logits_q = W_q · Block( h_t + (1/√Q) · Σ_{j<q} E_j(c_t^j) ) + b_q

where Block is a small RMSNorm → SwiGLU(×2) → RMSNorm residual body shared across all 16 layers. This means one trunk evaluation per frame and 16 cheap head evaluations, instead of 16 full autoregressive steps.

Why it's fast

1. 12.5 Hz is the headline number. One second of speech is 12.5 decoder steps. Codecs running at 50 Hz or 75 Hz need 4–6× more autoregressive steps for the same audio. Concretely:

A 10-second utterance is 125 recurrent steps. For comparison, a token-level LLM TTS at 50 Hz × 8 codebooks would be pushing ~500 trunk steps for the same clip.

2. O(1) state, not O(T) cache. The Mamba2 decoder carries a fixed (32 heads × 128 state × 64 dim) tensor plus a 3-frame conv window per layer. Generating 40 seconds costs exactly as much per step as generating 1 second — no attention matrix, no KV cache reallocation, no quadratic blowup. Long-form synthesis degrades gracefully instead of falling off a memory cliff.

3. Cross-attention K/V is computed once. Encoder output is projected to per-layer K/V a single time during prefill. Every subsequent frame does one small Q·Kᵀ against a fixed-length text sequence.

4. A shallow decoder. Only 6 decoder layers sit in the autoregressive loop. The 12-layer encoder runs exactly once, fully parallel over the input text.

5. The depth loop is cheap. The 16 codebook layers are resolved sequentially (layer q conditions on layers <q), but each step is one ×2 SwiGLU block at d=1024 — small enough that batch-1 generation is memory-bandwidth-bound rather than compute-bound.

How to Get Started

Setup

import torch
from transformers import AutoModelForSeq2SeqLM, PreTrainedTokenizerFast

repo = "telecomadm1145/Kiseki-TTS-1.1"
tok = PreTrainedTokenizerFast.from_pretrained(repo)
m = AutoModelForSeq2SeqLM.from_pretrained(repo, trust_remote_code=True).eval().cuda()

ids = m.build_tts_input_ids(tok.encode("こんにちは"), device="cuda")
out = m.generate_audio(ids, max_new_frames=250, temperature=0.9, top_k=50)
codes = out["audio_codes"][0][out["valid_mask"][0]]   # (T, Q)

build_tts_input_ids assembles [TTS] …text… <eos>. max_new_frames=250 ≈ 20 seconds of audio at 12.5 Hz.

Decode codes to waveform

import soundfile as sf
from qwen_tts import Qwen3TTSTokenizer

tokenizer = Qwen3TTSTokenizer.from_pretrained(
    "Qwen/Qwen3-TTS-Tokenizer-12Hz",
    device_map="cuda:0",
)

wavs, sr = tokenizer.decode({"audio_codes": codes})
sf.write("decode_output.wav", wavs[0], sr)

Sampling parameters

Generation stops when layer 0 emits EOS (2048). BOS and PAD are masked out of the logits, so they can never be sampled.