SpeechT5 TTS

Microsoft's SpeechT5 text-to-speech model with its HiFi-GAN vocoder, exported for loom.cpp: a text encoder and an autoregressive decoder that predicts mel-spectrogram frames, 16 kHz. Encodes text itself and has a built-in voice.

This is a loom.cpp export: a single self-describing GGUF that carries its own graph topologies, tokenizer (if any) and driver script, produced by loom-exporter.

Original model

Exported from microsoft/speecht5_tts. Weights are unmodified; this repo packages the same parameters into loom.cpp's GGUF format.

License

mit, inherited from the base model above.

Language(s)

en

Usage

Run it with loom-py -- loom-py-rt on PyPI:

pip install -U "loom-py-rt[hub]"

import loom

model = loom.Model.from_pretrained("loom-ai-org/speecht5-tts-loom")

# This model encodes text itself -- no phonemiser needed at all.
print(model.tokenizer)                       # kind, vocabulary size, default language

# sample_rate=16000: a rate is not something a checkpoint necessarily carries, so it is a value
# you have to know from the model's documentation and pass. It is used only if the GGUF declares none;
# a wrong rate does not fail, it plays the voice at the wrong speed.
audio = model.text2speech.infer("hello world", sample_rate=16000)
audio.save("out.wav")

# That uses the voice the file itself defaults to. Whether it carries others is under "Known
# limitations" (and, where it does, a section below says how to pick one).

Choosing a voice

The file carries one voice, slt (CMU ARCTIC, US female), and uses it when you name none. All seven CMU ARCTIC speakers ship in this repo as voice files under voices/ (slt.gguf is the same x-vector the file carries), and a name is enough -- it is fetched from this repo the first time:

print(model.voices)                          # the built-in one first, then the voice files

audio = model.text2speech.infer("Hello world.", voice="bdl")
audio.save("bdl.wav")

A SpeechT5 voice is a 512-number speaker embedding (an x-vector from SpeechBrain's spkrec-xvect-voxceleb), so any x-vector from that extractor works, including one of your own voice. Save it with numpy and convert it:

python -m loom_exporter.speecht5_voices microsoft/speecht5_tts -o voices \
    --from my_xvector.npy --name me --license "CC0-1.0"

Then pass voice="voices/me.gguf". The CMU ARCTIC recordings are "free for use for any purpose (commercial or otherwise)" provided Carnegie Mellon University's copyright notice is kept; every voice file records its licence and origin (loom.voice.license, loom.voice.origin).

The layer underneath

The call above is the high-level door: one per task, named for the modality pair it maps between, with the windowing, sampling and assembly this model needs already applied. Under it, model.infer(...) passes your arguments straight to the driver this GGUF embeds -- which is where you go for a knob the door does not name.

model.driver_source prints that driver, including a header comment documenting every argument it accepts for this model, and is the authority on it. See loom-py for the API and loom.cpp for what the engine does between the two.

Known limitations

One file holds both models. SpeechT5 predicts mel spectrograms; the waveform comes from microsoft/speecht5_hifigan, which this export carries in the same GGUF. Both are MIT.

Every frame depends on a random draw. SpeechT5 keeps its decoder prenet's dropout ON at inference (a Tacotron 2 choice). loom-py seeds it with 0 unless you pass seed, so the same call gives the same audio and another seed gives another take. Verified against transformers' generate_speech with the draws pinned: the same number of frames and a waveform within 6e-06 rms.

Numbers are spelled out for you ($15,000.50 is read "fifteen thousand point five zero dollars"): the vocabulary has no digits, so the file carries the reference's own number normaliser (SpeechT5Tokenizer(normalize=True)), which loom runs before tokenizing. It needs loom 1.0.0-rc12 or later; on rc11 and earlier a number reaches the model as an unknown token, as it does upstream by default.

English only, trained on LibriTTS. Stops on the model's own stop head, at most ten decoder steps (20 frames, 0.32 s) per input character.

Files

• speecht5-tts.gguf -- the model, exported with loom-exporter.
• voices/*.gguf -- the seven CMU ARCTIC speakers' x-vectors as loom voice files, from Matthijs/cmu-arctic-xvectors (each speaker's arctic_a0508 utterance) by loom_exporter.speecht5_voices. Only needed to pick a voice other than the built-in slt.