WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS
WordVoice is a novel speech generation framework that transforms traditional implicit end-to-end TTS generation into an explicit, highly controllable paradigm. Built on top of the CosyVoice3 framework, it enables precise and decoupled word-level control over five acoustic dimensions.
- Paper: WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS
- GitHub Repository: XXH333/WordVoice-main
- Project Page & Demo: WordVoice Demo
✨ Features
🎯 Explicit Word-Level Control
Supports independent and decoupled control of five acoustic attributes for each input word:
- ⏱️ Duration: Word-level pronunciation duration.
- ⏸️ Boundary: 5-level pause classification (
b0–b4). - 🔊 Energy: Word-level volume/loudness (
0–1). - 🎵 Pitch: Word-level core fundamental frequency (
-1–1). - 📈 Tone: 7 categories of prosodic morphologies (flat, rise, strong rise, fall, strong fall, peak, valley).
🧠 "Acoustic Thinking" Mechanism via Bound-Token
Employs a bound-token (<b>) mechanism within the autoregressive (AR) language model. Before generating the speech tokens for a specific word, the model explicitly predicts its acoustic attributes, realizing an intelligent process of "planning prosody first, then generating sound."
🛠️ Quick Start
Installation
We recommend using Conda to manage your Python environment.
conda create -n wordvoice python=3.10 -y
conda activate wordvoice
git clone https://github.com/XXH333/WordVoice-main.git
cd WordVoice-main
pip install -e .
pip install num2words==0.5.14 x_transformers==2.11.24
Download Model Weights
Run the following script to automatically download the pre-trained weights and dependencies (such as CosyVoice3, MMS-FA, etc.):
bash download_models.sh
Inference
You can run the out-of-the-box inference script to experience both the Free Mode and Control Mode of WordVoice:
python wordvoice_infer.py
For custom prompts and detailed control parameters, refer to wordvoice_infer.py and the GitHub repository.
📝 Citation
If you find this work or the models useful, please cite:
@misc{nie2026wordvoice,
title={WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS},
author={Sihang Nie and Jinxin Ji and Xiaofen Xing and Deyi Tuo and Chengbin Jin and Jialong Mai and Xiangmin Xu},
year={2026},
eprint={2607.06461},
archivePrefix={arXiv},
primaryClass={eess.AS},
url={https://arxiv.org/abs/2607.06461},
}