XTTS-v2 Yoruba (OpenBibleTTS 8-Epoch Fine-Tune)

This repository provides fine-tuned weights, extended tokenizer vocabulary, and configuration files for Coqui XTTS-v2 fine-tuned on Yoruba (yo) speech from the OpenBible dataset.

The model extends XTTS-v2's multilingual voice-cloning capabilities to Yoruba, maintaining conditioning latents, speaker cloning fidelity, and cross-lingual transfer from a reference audio clip (3 to 10 seconds).

Model Summary

• Base Architecture: Coqui XTTS-v2 (Autoregressive GPT text/mel encoder + Discrete VAE spectrogram vocoder)
• Parameters: 521.8M
• Target Language: Yoruba (yo)
• Tokenizer: Extended Byte Pair Encoding (BPE) with special language tag [yo] (extended vocabulary: 2,000 tokens)
• Dataset: multilingual-tts/open-bible (Yoruba split)
• Training Samples: 8,000 paired audio-text samples (resampled to 22,050 Hz)
• Epochs: 8 full epochs (31,360 total optimization steps)
• Hardware: NVIDIA A10G (24 GB VRAM)

Training Convergence & Metrics

The model trained steadily across all 8 epochs with AdamW and MultiStep scheduling (learning rate = 5e-6, effective batch size = 8):

Epoch Progression Table

Key Metric Takeaways

• Text Cross-Entropy Loss decreased from 0.2430 to 0.0323 (86.7% reduction), confirming solid phonetic alignment to Yoruba tonal marks and orthography.
• Mel Spectrogram Loss converged from 3.9983 to 2.1748 (45.6% reduction), sharpening voice clarity and acoustic fidelity.
• Validation Loss showed consistent, monotonic decrease across all 8 epochs (from 2.6773 down to 2.2072).

Audio Comparison: Original vs. XTTS Cloned Synthesis

Below is a side-by-side comparison between the reference speakers and the speech synthesized by this fine-tuned model.

All three reference audios are sourced from the benjaminogbonna/nigerian_common_voice_dataset (Yoruba split) by Benjamin Ogbonna:

• Samples 1 & 2: Zero-shot voice cloning reproducing each speaker's own voice and words.
• Sample 3: Cross-speaker voice transfer, synthesizing speech from a distinct OpenBible training narrator voice on an utterance spoken by a Ben Ogbonna Common Voice speaker.

Sample 1: "Ìjọba Òǹdo ní òun yóò gbé ibùdó ìgbókùsíí kúrò nílé ìwòsàn."

Sample 2: "Ṣèyí ń ṣe bíi Baṣọ̀run Gáà, tí kò bọ̀wọ̀ fún òfin."

Sample 3: "Makinde ní kò sí ọwọ́jà ààrùn àìmọ̀ ẹlẹ́ẹ̀kejì ní ìpínlẹ̀ Ọ̀yọ́."

Repository Files

This repository contains all files necessary to run standalone synthesis or merge with standard Coqui pipelines:

How to Use

Ensure coqui-tts and dependencies are installed:

pip install coqui-tts "transformers<4.45.0" torch torchaudio huggingface_hub

Load the model and synthesize speech directly from this Hugging Face repository:

import os
import re
import torch
import torchaudio
from huggingface_hub import snapshot_download
from TTS.tts.configs.xtts_config import XttsConfig
from TTS.tts.models.xtts import Xtts

# 1. Download model repository
model_dir = snapshot_download(repo_id="samuelolubukun/xtts-v2-yoruba-openbible")

# 2. Load model config and checkpoint
config = XttsConfig()
config.load_json(os.path.join(model_dir, "config.json"))
model = Xtts.init_from_config(config)
model.load_checkpoint(
    config,
    checkpoint_path=os.path.join(model_dir, "model.pth"),
    vocab_path=os.path.join(model_dir, "vocab.json"),
    use_deepspeed=False
)
model.cuda() if torch.cuda.is_available() else model.cpu()

# 3. Text cleaner fallback for custom language support
orig_preprocess = model.tokenizer.preprocess_text
def safe_preprocess_text(txt, lang):
    try:
        return orig_preprocess(txt, lang)
    except NotImplementedError:
        txt = txt.lower()
        return re.sub(r"\s+", " ", txt).strip()
model.tokenizer.preprocess_text = safe_preprocess_text

# 4. Compute speaker conditioning latents (using bundled reference audio or your own 3-10s WAV)
ref_audio = os.path.join(model_dir, "samples/sample1_original.mp3")
gpt_cond_latent, speaker_embedding = model.get_conditioning_latents(
    audio_path=ref_audio,
    gpt_cond_len=30,
    max_ref_length=30,
    sound_norm_refs=False,
)

# 5. Synthesize speech
text = "Ẹ n lẹ́ o, báwo ni gbogbo yín ṣe wà? Èkó dára púpọ̀."
out = model.inference(
    text=text,
    language="yo",
    gpt_cond_latent=gpt_cond_latent,
    speaker_embedding=speaker_embedding,
    temperature=0.3,
    length_penalty=1.0,
    repetition_penalty=5.0,
    top_k=30,
    top_p=0.85,
)

# 6. Save output audio (24,000 Hz)
torchaudio.save("output_yo.wav", torch.tensor(out["wav"]).unsqueeze(0), 24000)
print("Saved synthesized speech to output_yo.wav")

Limitations and Ethical Usage

• Domain & Register Bias: Trained on read, scripture-domain speech characterized by a narrow theological vocabulary and a formal liturgical register. As a consequence, the model may not transfer out-of-the-box with natural prosody to informal or fast-paced conversational speech.
• Dataset Labels: Speaker labels and forced segment alignments in the upstream OpenBibleTTS dataset were generated automatically and may contain slight boundary inconsistencies.
• Ethical Voice Cloning: This model is intended for research, linguistic preservation, and accessibility in African languages. Explicit consent should be obtained prior to cloning recognizable individual voices.
• Licensing: Inherits the Coqui Public Model License / MRCUL.

Author & Attribution

Developed and fine-tuned by Samuel Olubukun:

• Website: samuelolubukun.com
• GitHub: @samolubukun
• Hugging Face: @samuelolubukun

Citation

If you use this model or checkpoints in your research, applications, or benchmarks, please cite:

@misc{olubukun2026xttsyoruba,
  author = {Samuel Olubukun},
  title = {XTTS-v2 Yoruba: 8-Epoch Fine-Tuning on OpenBibleTTS},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/samuelolubukun/xtts-v2-yoruba-openbible}}
}