speecht5_tts-pld-ceb-v2
sapinsapin/speecht5_tts-pld-ceb
continue-finetuned on every usable Cebuano clip in
sapinsapin/pld — 14,007 of
them, against the ~1,800 the published checkpoint saw.
It was never a shortage of data. PLD holds ~50,900 Cebuano rows; 15,163 pass the
TTS filter (read speech, not a prompt, ≥3 words) and 14,007 survive the digit
and length caps — 19 hours. The original run stopped at 1,847 because
--max-samples 3000 was a default nobody revisited. This gives the same model
the rest of its own language.
"Continue", not "resume": the published repo ships model.safetensors and
training_args.bin and nothing else — no optimizer moments, no scheduler state
— so this warm-starts from the released weights with a fresh optimizer and a
fresh linear schedule.
| training clips | 13,807 train + 200 eval (14,007 total) |
| steps | 4,000 (≈9 epochs) |
| batch | 8 × 4 accumulation |
| learning rate | 1e-05 |
| precision | fp32 + gradient checkpointing (fp16 NaNs SpeechT5's mel loss) |
| eval loss | 0.3687, against 0.4070 for the base checkpoint |
| hardware | one free-tier Colab T4, ~3 h |
What it bought
There is no way to measure a synthetic voice directly, so the test is indirect: play each clip to a speech recognizer that never saw the original sentence, and count how much of the text it fails to recover. Character error rate (CER) is the share of characters it gets wrong. Lower is better. This measures whether speech is intelligible to a machine listener — not whether it sounds natural or authentically Cebuano, which still needs human ears.
Ten held-out pld test lines, transcribed by whisper-large-v3-turbo:
| base | v2 | |
|---|---|---|
CER, arctic:slt | 0.141 | 0.093 |
WER, arctic:slt | 0.497 | 0.388 |
CER, native pld:ceb:12 | 0.210 | 0.148 |
WER, native pld:ceb:12 | 0.637 | 0.524 |
A third of the character errors, gone. At 0.093 this scores below mms-tts-ceb
on the same lines (0.153).
Four listeners, not one
A single judge is a single opinion, and every recognizer is deaf in its own way. Whisper has no Cebuano in its training data at all, so some share of every score above is the judge's failure rather than the model's — a real Cebuano speaker reading these same lines scores 0.081, not 0. Rescoring the same audio with four different recognizers separates the two:
| judge | base | v2 | what it is |
|---|---|---|---|
sapinsapin/whisper-small-pld-ceb | 0.139 | 0.065 | small Whisper, finetuned on Cebuano |
whisper-large-v3-turbo | 0.146 | 0.123 | large, zero-shot, no Cebuano |
whisper-turbo + Filipino LoRA | 0.126 | 0.092 | large, nudged toward Philippine speech |
facebook/mms-1b-all (ceb adapter) | 0.125 | 0.067 | CTC model, 491k hours, ceb head |
CER, arctic:slt, ten lines. This is a different bench run from the table
above — which is why turbo reads 0.123 here and 0.093 there, with no change to
the weights. See the determinism note at the bottom; it is the same lesson.
The pattern is the finding. The judge that actually knows Cebuano sees the retrain cut errors by 53%; the judge that does not sees 16%. Zero-shot Whisper is measuring partly its own ignorance, and that noise compresses the distance between a good model and a poor one. Read down a column, never across: a CTC model and a sequence-to-sequence model make different kinds of mistakes, so their absolute numbers are not on the same scale.
One caveat on the in-domain judge: whisper-small-pld-ceb was finetuned on the
same corpus this model trained on. It is the most informed listener available,
and also the most likely to reward speech that sounds specifically like PLD.
Treat it as the sharpest instrument, not the neutral one.
What it did not fix
Speaker conditioning. Handed a native Cebuano x-vector instead of the American
arctic:slt, the decoder still collapses on most voices: 3 of 16 native
candidates survive, against 1 of 16 for the base checkpoint. Tripled, still 13
of 16 broken — one renders a 4-second line as 17 seconds at rms 0.003.
Seven-fold data was not the fix, because the bottleneck is not clip count but clip distribution: those 19 hours are 139 speakers, and the best-covered has 15.5 minutes. PLD is broad and shallow — built for ASR coverage, not TTS depth. Averaging harder over 139 people does not produce one person.
If you need a single reliable voice, use
Splintir/speecht5_tts-pld-ceb-solo,
which continue-trains this checkpoint on one speaker alone.
Usage
SpeechT5 holds no voice of its own — every call needs a 512-d x-vector. None
ships with the original PLD checkpoints, which is why they are usually run with
an American speaker from the HuggingFace tutorial. speaker.npy in this repo
is a real ceb speaker from the training data, so you can skip that.
import numpy as np, torch, soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import SpeechT5ForTextToSpeech, SpeechT5HifiGan, SpeechT5Processor
REPO = "Splintir/speecht5_tts-pld-ceb-v2"
processor = SpeechT5Processor.from_pretrained(REPO)
model = SpeechT5ForTextToSpeech.from_pretrained(REPO).eval()
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan").eval()
# speaker.npy is stored as (512,); the model wants (1, 512). Without the
# unsqueeze this raises "The first dimension of speaker_embeddings must be
# either 1 or the same as batch size".
speaker = torch.from_numpy(np.load(hf_hub_download(REPO, "speaker.npy"))).float().unsqueeze(0)
ids = processor(text="Maayong buntag sa imong tanan.", return_tensors="pt")["input_ids"]
with torch.inference_mode():
speech = model.generate_speech(ids, speaker, vocoder=vocoder)
sf.write("out.wav", speech.numpy(), 16000) # 16 kHz mono
# In a notebook (Colab, Jupyter), play it inline instead of saving:
from IPython.display import Audio, display
display(Audio(speech.numpy(), rate=16000))
Runs unmodified on a stock Colab CPU runtime — every dependency is preinstalled,
including the sentencepiece the tokenizer needs — at roughly 1.8× real time.
For GPU, .to("cuda") the model, the vocoder and ids/speaker, then .cpu()
the result before sf.write.
The display(Audio(...)) line renders a play button in a notebook cell. It
produces no sound over a terminal or SSH session; there, write the wav and fetch
it (from google.colab import files; files.download("out.wav")).
Spell numbers out: the tokenizer is character-level Latin and drops digits silently.
Two things to know before trusting a number
Generation is not deterministic. SpeechT5's decoder prenet keeps dropout
active during inference, by design, as Tacotron2's does. Five renders of one
line from one checkpoint gave five different durations (3.17–3.30 s) and rms
0.062–0.074. This checkpoint scored CER 0.093 in one bench run and 0.123 in
another with nothing changed — so compare systems within a run, never across
runs. In production, cache audio by hash(text + voice) to freeze one
known-good render.
An out-of-distribution x-vector produces 12 seconds of quiet mumbling, not an error. If output is long and near-silent, the speaker embedding is the suspect, not the text.
Trained with scripts/train_tts.py;
preprocessing matches finetune_tts.py from the
halohalo pipeline — one x-vector per
clip, never averaged.