OmniVoice Word-Control
OmniVoice (0.6B masked-diffusion LM TTS on Qwen3-0.6B) fine-tuned for five-dimensional word-level acoustic control, following the task of WordVoice (arXiv:2607.06461) and its dataset WordVoice-5A.
Control is injected as inline special tokens placed before any word (any subset works; untagged words are planned freely by the model):
<|dur_9|><|bnd_0|><|eng_15|><|pit_17|><|ton_rise|>word
| tokens | attribute | discretization |
|---|---|---|
<|dur_0..63|> | word duration | 40 ms frames (40 ms – 2.56 s) |
<|bnd_0..4|> | pause after word | b0 (none) … b4 (long, >0.4 s) |
<|eng_0..19|> | energy | 20 uniform bins over [0, 1] |
<|pit_0..19|> | pitch (core F0) | 20 uniform bins over [−1, 1] |
<|ton_flat|rise|rrise|fall|ffall|peak|valley|> | pitch contour | 7 morphologies |
Usage
from omnivoice import OmniVoice # code: github.com/k2-fsa/OmniVoice
model = OmniVoice.from_pretrained("multimodalart/omnivoice-word-control", device_map="cuda")
audio = model.generate(
text="I will <|pit_3|><|ton_ffall|>never agree to <|bnd_4|>this!",
language="en",
ref_audio="ref.wav",
ref_text="Transcript of the reference clip.",
)
The tokenizer in this repo already contains the 116 control tokens. For strict overall
timing, also pass duration= (the diffusion canvas is fixed before generation, so word
duration tokens redistribute time within it). Try it in the
demo Space.
Training
- Init from
k2-fsa/OmniVoice; all parameters trained; masked-diffusion objective unchanged (control tokens are conditioning only — no loss on text). - Data: WordVoice-5A English, all 4 train shards (~2,138 h, 1.18 M utterances); on-the-fly tag construction with dropout (15% utterances untagged / 20% words / attributes kept @ 0.7) to preserve plain TTS and enable partial control.
- 40k steps × 8,192 effective tokens (~1 epoch), lr 5e-5 cosine, bf16, single A100-40GB, ≈6 h.
- Final eval loss 4.109 (best of run). Directional probes (hi/lo tag ratio, seed-matched): pitch 3.61, energy 2.52, duration 2.70.
Fine-tuning recipe: the word-control patch set on the OmniVoice trainer
(omnivoice/data/word_control.py — tag vocabulary, binning, alignment, dropout).
License & lineage
Weights are CC-BY-NC-4.0 (derivative of the CC-BY-NC OmniVoice release). Data: WordVoice-5A (CC-BY-4.0). Task formulation: WordVoice, arXiv:2607.06461.