kokoro-inno-clone-tuner

pip install inno-kokoro

Zero-shot voice tuner for Kokoro-82M.

Outputs base Kokoro compatible voice packs @ [510, 1, 256].

Samples

Same passage for every voice, enrolled from the references. LibriTTS-R speakers are dev-clean held out from training.

Usage

Code available on Github: remsky/inno-kokoro.

Integrated into Kokoro-FastAPI (v0.9.0+)

pip install inno-kokoro

Enrolling a voice:

from inno_kokoro.enroll import Tuner, enroll, read
from kokoro import KPipeline

tuner = Tuner()  # pulls model.safetensors into the huggingface cache
pack, _ = enroll(*read("my_ref.wav"), tuner)

pipe = KPipeline(lang_code="a")
wav = next(pipe("Hello from a tuned voice.", voice=pack)).audio

The pack is a plain tensor; torch.save(pack, "voices/am_me.pt") makes it a voice file like any other, prefixed by accent and gender like the stock packs.

The pitch-tracking ceiling is set automatically from the reference's harmonic spacing, so band-limited or archival sources land in the right octave without tuning.

• enroll(..., fmax=180) overrides it if a voice still reads the wrong register.

Architecture

Enrollment embeds an input audio sample via the baked speaker encoder and maps through model.safetensors to build the encoder side; prosody aspects are measured to build the predictor half. Everything loads from model.safetensors, no extra downloads:

• speaker encoder: a 6.6M-param WeSpeaker ResNet34 distilled from UniSpeech-SAT-sv embeddings
  • matches within 0.002 speaker cosine on hold-out dev-clean (380mb fp32 -> 14mb fp16)
• style head: the decoder (timbre) half of the pack comes from the speaker embedding (512) mapped to Kokoro style (256). Shifts along a learned spectral-tilt direction by the reference's measured tilt.
• prosody blend: the predictor (prosody) half uses the stock Kokoro packs as a foundation:
  • Blended by nonnegative least squares to the reference F0 mean, F0 spread and syllable rate.
  • Packs more than 4 st from the reference's pitch are excluded.
  • Higher graded packs are weighted as preferred during selection.
• prosody head (v0.2): a linear adjust on the blend from the reference's F0 mean and spread,
  • fitted to deltas optimized through the frozen predictor. enroll(..., head=False) skips it.

Enrollment runs in about 0.05 s per second of reference on CPU (a 30 s reference in 1.4 s) and 0.1-0.3 s total on a GPU, after the one-time model load.

Total model size @ fp16: ~24mb

Limits

• English trained (LibriTTS-R) currently
• Some texture/identity is traded to maintain audio quality and stability within Kokoro's range.
• Rythym is a best-effort match to the speaker.
  • Prosody is a convex blend of stock prosody by pitch, spread and pace. Outside of that range, it will only pin to the nearest edge
• Reference clip requirements:
  • 3 second minimum, up to a max of 30s
  • Reasonably free of audio artifacts
  • Single speaker

Training and Metrics

All stages: Kokoro frozen; LibriTTS-R train-clean-100 unless noted.

Trained via HF Jobs a10g-small; under $20 in GPU time:

• Stage 1: mapped speaker embedding to timbre:
  • Style head on a speaker-embedding cycle loss: clone, render, re-embed, cosine to the input audio embedding.
  • [10k steps: dev cosine ~0.92; nearest stock voice baseline 0.857]
• Stage 2: the embedding alone missed texture and pitch; adjustments:
  • Augmented pitch and speed. Added losses via a WavLM layer-statistics L1, a small log-F0 error, and spectral-tilt.
  • [11k steps: dev cosine 0.934, DNSMOS 3.30]
• Stage 3: tuned the encoder side to match the blended predictors:
  • Predictor half fixed to the blended rows, only the decoder half trained.
  • [3k steps: dev cosine 0.932, DNSMOS 3.48]
• Stage 4: distilled the speaker encoder so enrollment needs no UniSpeech download:
  • WeSpeaker ResNet34 plus a projection, trained on 1 - cos(student, teacher) over LibriTTS-R crops with synthetic channel damage; the style head is untouched.
  • [20k steps: hold-out dev-clean parity, DNSMOS equal; archival references sat about 0.05 behind the teacher]
• Stage 5: the student lagged the teacher on archival references; synthetic damage had plateaued:
  • Mixed real public-domain archival broadcast audio into the distillation crops.
  • [8k steps: archival cosine up 0.03-0.05, DNSMOS unchanged]

• Stage 1: the blend under-predicted pitch range and sentence-final falls:
  • Fitted a predictor-half delta per speaker (426; adds VoxPopuli-en and Emilia-YODAS EN)
  • Targeted F0 mean, spread, IQR, pause fraction and speech time.
  • Ridge head from reference F0 mean + spread to the fitted delta.
  • [hold-out: F0 mean error 0.9 -> 0.45 st, spread 0.6 -> 0.35 st, IQR 0.9 -> 0.5 st]
• Stage 2: the student trailed the teacher on public-speaking and in-the-wild references:
  • Added VoxPopuli and Emilia speakers to the distillation crops.
  • [8k steps: hold-out cosine VoxPopuli 0.976 -> 0.986, Emilia 0.964 -> 0.983; WavLM cosine 0.941 -> 0.944, DNSMOS unchanged]
• Stage 3: the timbre half had only seen LibriTTS speakers:
  • Repeated v0.1.0 stage 3 with VoxPopuli and Emilia speakers in the draw.
  • [3k steps: WavLM cosine samples 0.927 -> 0.935, dev-clean 0.956 -> 0.959, VoxPopuli 0.926 -> 0.939, Emilia 0.931 -> 0.936; DNSMOS unchanged]

Measured on the shipped weights through the stock KPipeline: one fixed passage rendered for the seven sample voices plus ten hold-out dev-clean speakers, F0 by Praat over 6 s chunks with the reference's tracking ceiling applied to both sides.

The head's gain is spread; on F0 mean it scatters more per speaker and flips the signed bias from -0.2 to +0.3 st.

Zero-shot identity benchmark

LibriSpeech test-clean, F5-TTS cross-sentence split (1127 utterances, 39 held-out speakers). Reference in, new sentence out, scored against the speaker's real recording.

• normalized: how far the render sits between sounding like a stranger (0) and a second real take of the speaker (1)
• RTF: RTX 4060 Ti

Inno gets about a third of the identity, twice as close as the nearest stock pack (with UTMOS intact and keeping Kokoro's stable natural style).

StyleTTS2 is the same architecture with an unfrozen decoder, assumed to be the ceiling of this style space.

• StyleTTS2: alpha 0.3, beta 0.7, 5 steps.
• OpenVoice v2: MeloTTS EN-US, tau 0.3.
• F5-TTS: 32 NFE, CFG 2.

Licenses and credits

Adapter weights and code: Apache-2.0, except the baked speaker encoder (enc.* in model.safetensors), which is CC BY-SA 3.0 (see below)

• Built on hexgrad/Kokoro-82M (Apache-2.0, StyleTTS2 architecture)
• The speaker encoder is a WeSpeaker ResNet34 (code Apache-2.0)
  • initialised from the VoxCeleb-trained ResNet34-LM weights (CC BY 4.0)
  • distilled from microsoft/unispeech-sat-base-plus-sv
    • (Microsoft UniSpeech, CC BY-SA 3.0 on GitHub); encoder weights under CC BY-SA 3.0
• Training data (no audio redistributed beyond the samples):
  • LibriTTS-R (CC BY 4.0), also the LibriTTS sample references
  • VoxPopuli (CC0)
  • Emilia-YODAS EN (CC BY 4.0)
• Sample references:
  • Amelia Earhart from "Speech by Amelia Earhart", Library of Congress AFC 1986/022
  • Jane Goodall from BBC Woman's Hour excerpt, BBC, CC BY 3.0
  • Vincent Price public domain from "The House On Haunted Hill"

Only clone voices you have permission to clone, even the shallow cloning offered via this model and technique.