speecht5_tts-pld-hil-v2
sapinsapin/speecht5_tts-pld-hil continue-finetuned on 10,664 hil clips from sapinsapin/pld -- the language's full usable set, not a sample of it.
The published checkpoints were each trained on ~1,800 clips for 1,000 steps, a cap that was a default rather than a measurement of how much data existed. This run keeps the same recipe and gives the model the rest of its own language.
| training clips | 10,464 train + 200 eval |
| steps | 4,000 |
| batch | 8 x 4 accumulation |
| learning rate | 1e-05 |
| precision | fp32 + gradient checkpointing (fp16 NaNs SpeechT5's mel loss) |
| eval loss (this run) | 0.3681 |
Usage
SpeechT5 holds no voice of its own -- every call needs a 512-d x-vector. None
ships with the original PLD checkpoints, which is why they are usually run with
an American speaker from the HuggingFace tutorial. speaker.npy in this repo is a real hil speaker from the training data, so you can skip that.
import numpy as np, torch, soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import SpeechT5ForTextToSpeech, SpeechT5HifiGan, SpeechT5Processor
REPO = "Splintir/speecht5_tts-pld-hil-v2"
processor = SpeechT5Processor.from_pretrained(REPO)
model = SpeechT5ForTextToSpeech.from_pretrained(REPO).eval()
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan").eval()
# speaker.npy is stored as (512,); the model wants (1, 512). Without the
# unsqueeze this raises "The first dimension of speaker_embeddings must be
# either 1 or the same as batch size".
speaker = torch.from_numpy(np.load(hf_hub_download(REPO, "speaker.npy"))).float().unsqueeze(0)
ids = processor(text="Maayong buntag sa imong tanan.", return_tensors="pt")["input_ids"]
with torch.inference_mode():
speech = model.generate_speech(ids, speaker, vocoder=vocoder)
sf.write("out.wav", speech.numpy(), 16000) # 16 kHz mono
# In a notebook (Colab, Jupyter), play it inline instead of saving:
from IPython.display import Audio, display
display(Audio(speech.numpy(), rate=16000))
Runs unmodified on a stock Colab CPU runtime -- every dependency is
preinstalled, including the sentencepiece the tokenizer needs -- at roughly
1.8x real time. For GPU, .to("cuda") the model, the vocoder and
ids/speaker, then .cpu() the result before sf.write.
The display(Audio(...)) line renders a play button in a notebook cell. It
produces no sound over a terminal or SSH session; there, write the wav and fetch
it (from google.colab import files; files.download("out.wav")).
Spell numbers out: the tokenizer is character-level Latin and drops digits silently.
Trained with scripts/train_tts.py;
preprocessing matches finetune_tts.py from the
halohalo pipeline -- one x-vector per
clip, never averaged.