KaniTTS Finetuned
Text-to-Speech (TTS) model designed for high-speed, high-fidelity audio generation.
KaniTTS is built on a novel architecture that combines a powerful language model with a highly efficient audio codec, enabling it to deliver exceptional performance for real-time applications.
Model Details
KaniTTS operates on a two-stage pipeline, leveraging a large foundation model for token generation and a compact, efficient codec for waveform synthesis.
The two-stage design of KaniTTS provides a significant advantage in terms of speed and efficiency. The backbone LLM generates a compressed token representation, which is then rapidly expanded into an audio waveform by the NanoCodec. This architecture bypasses the computational overhead associated with generating waveforms directly from large-scale language models, resulting in extremely low latency.
Features
This model trained primarily on English for robust core capabilities and the tokenizer supports these languages: English, Arabic, Chinese, French, German, Japanese, Korean, and Spanish.
The base model can be continually pretrained on the multilingual dataset producing high-fidelity audio at sample rates 22kHz.
This model powers voice interactions in the modern agentic systems, enabling seamless, human-like conversations.
- Model Size: 450M parameters (pretrained version)
- License: Apache 2.0
Examples
| Text | Audio |
|---|---|
| I do believe Marsellus Wallace, MY husband, YOUR boss, told you to take me out and do WHATEVER I WANTED. | |
| What do we say the the god of death? Not today! | |
| What do you call a lawyer with an IQ of 60? Your honor | |
| You mean, let me understand this cause, you know maybe it's me, it's a little fucked up maybe, but I'm funny how, I mean funny like I'm a clown, I amuse you? I make you laugh, I'm here to fucking amuse you? |
Sources
- Website: nineninesix.ai
- GitHub Repo: https://github.com/nineninesix-ai/kani-tts
- Base Model Card on HF: nineninesix/kani-tts-450m-0.1-pt
- FT Model Card on HuggingFace: nineninesix/kani-tts-450m-0.2-ft
- Link to HF Space: nineninesix/KaniTTS
- Inference Example: Colab Notebook
- Finetuning Example: Colab Notebook
- Example Dataset for Fine-tuning: Expresso Conversational
- Waiting List for Pro Version
Recommended Uses
- Conversational AI: Integrate into chatbots, virtual assistants, or voice-enabled apps for real-time speech output.
- Edge and Server Deployment: Optimized for low-latency inference on edge devices or affordable servers, enabling scalable, resource-efficient voice applications.
- Accessibility Tools: Support screen readers or language learning apps with expressive prosody.
- Research: Fine-tune for domain-specific voices (e.g., accents, emotions) or benchmark against other TTS systems.
Limitations
- Performance may vary with finetuned variants, long inputs ( > 2000 tokens) or rare languages/accents.
- Emotion control is basic; advanced expressivity requires fine-tuning.
- Trained on public datasets; may inherit biases in prosody or pronunciation from training data.
Training Data
- Dataset: Curated from LibriTTS, Common Voice and Emilia (~50k hours).
- Pretrained mostly on English speech for robust core capabilities, with multilingual fine-tuning for supported languages.
- Metrics: MOS (Mean Opinion Score) 4.3/5 for naturalness; WER (Word Error Rate) < 5% on benchmark texts.
- Hardware: Pretrained on 8x H200 over 8 hours.
Inference on Nvidia RTX 5080:
- Latency: < 1s to generate 15 seconds of audio
- Memory Usage: 2GB GPU VRAM
This performance makes KaniTTS suitable for real-time conversational AI applications and low-latency voice synthesis.
Tips & Tricks
- Language Optimization: For the best results in non-English languages, continually pretrain this model on datasets from your desired language set to improve prosody, accents, and pronunciation accuracy. Additionally, finetune NanoCodec for desired set of languages.
- Batch Processing: For high-throughput applications, process texts in batches of 8-16 to leverage parallel computation, reducing per-sample latency.
- Blackwell GPU Optimization: This model runs efficiently on NVIDIA's Blackwell architecture GPUs for faster inference and reduced latency in real-time applications.
Credits
- This project was inspired by the works of Orpheus TTS and Sesame CSM.
- It utilizes the LiquidAI LFM2 350M as its core backbone and
- Nvidia NanoCodec for efficient audio processing.
Responsible Use and Prohibited Activities
The model is designed for ethical and responsible use. The following activities are strictly prohibited:
- The model may not be used for any illegal purposes or to create content that is harmful, threatening, defamatory, or obscene. This includes, but is not limited to, the generation of hate speech, harassment, or incitement of violence.
- You may not use the model to generate or disseminate false or misleading information. This includes creating deceptive audio content that impersonates individuals without their consent or misrepresents facts.
- The model is not to be used for any malicious activities, such as spamming, phishing, or the creation of content intended to deceive or defraud.
By using this model, you agree to abide by these restrictions and all applicable laws and regulations.
Contact
Have a question, feedback, or need support? Please fill out our contact form and we'll get back to you as soon as possible.