XTTS-v2 Yoruba Conversational (Common Voice Domain-Adapted)

This repository provides fine-tuned weights, extended tokenizer vocabulary, and configuration files for Coqui XTTS-v2 adapted for conversational Yoruba (yo).

This model represents Stage 2 in a two-stage training strategy:

1. Stage 1 (Foundation): samuelolubukun/xtts-v2-yoruba-openbible — 8-epoch foundation model trained on high-fidelity OpenBible audio to master Yoruba orthography, diacritics (ẹ, ọ, ṣ), tone markers (High/Mid/Low), and stable phonetics.
2. Stage 2 (Conversational Adaptation — This Model): Adapted from the Stage 1 checkpoint on benjaminogbonna/nigerian_common_voice_dataset (Yoruba split) across 5 full epochs using a conservative learning rate (1.5e-6) to break the rigid, dragging scripture recitation cadence and internalize fluid colloquial co-articulation without catastrophic forgetting.

Model Summary

• Base Architecture: Coqui XTTS-v2 (Autoregressive GPT text/mel encoder + Discrete VAE spectrogram vocoder)
• Base Model Checkpoint: samuelolubukun/xtts-v2-yoruba-openbible (31,360 steps)
• Parameters: 521.8M
• Target Language: Yoruba (yo)
• Tokenizer: Extended Byte Pair Encoding (BPE) with special language tag [yo] (extended vocabulary: 9,987 tokens)
• Adaptation Dataset: benjaminogbonna/nigerian_common_voice_dataset (Yoruba split)
• Filtered Training Set: 2,367 clean conversational audio samples (filtered for duration, quality, and valid diacritics)
• Adaptation Steps: 7,895 optimization steps (5 epochs, lr=1.5e-6, grad_accum=4, batch_size=2)
• Hardware: NVIDIA A10G (24 GB VRAM)

Conversational Adaptation Dashboard & Convergence

The visualization below highlights the multidimensional shift from the Stage 1 OpenBible foundation model to this Stage 2 Common Voice conversational model across cadence, pitch dynamics, acoustic loss, and syllable co-articulation:

Training Progression Summary

Acoustic Improvements Over Stage 1

Objective acoustic and linguistic evaluation across unseen Common Voice test speakers demonstrates major improvements:

Audio Comparison: Original vs. Stage 1 vs. Stage 2 Cloned Synthesis

Below is a side-by-side comparison between the unseen reference speakers and the speech synthesized by the Stage 1 foundation model versus this Stage 2 conversational adapted model.

All reference audios are sourced from benjaminogbonna/nigerian_common_voice_dataset (Yoruba split) by Benjamin Ogbonna:

Sample 1 (Complex Clause): "Àgbèrè, ìpànìyàn, ṣíṣẹ́yún, olè, àìlóòótó, àìní-ìtẹríba jẹ́ ìwà burúkú."

Sample 2 (Dialogue Question): "Èkó dára ju Àbújá lọ, àbí bẹ́ẹ̀ kọ́?"

Sample 3 (Relative Clause): "Ìwé tí mo ní kí o kà lọ́jọ́sí, wọ́n ti ṣe é sínu fọ́nrán."

Sample 4 (Compound Narrative): "Wọ́n yàn olórí ìjọ, àmọ́ orí gbé e dé ipò igbákejì ààrẹ."

Sample 5 (Expressive Statement): "Gbogbo àwọn dọ́kítà ti pinu láti wá ọ̀nà àbáyọ sí ààrùn."

🔁 Stage 1 OpenBible Benchmark Samples: Foundation vs. Conversational Comparison

These are the exact benchmark samples originally evaluated on the samuelolubukun/xtts-v2-yoruba-openbible Stage 1 foundation model repository. Below, the same reference audio clips and prompt texts are synthesized side-by-side using the Stage 2 Conversational fine-tuned model to directly compare the change in cadence, natural pacing, and conversational tone:

Stage 1 Benchmark Sample 1: "Ìjọba Òǹdo ní òun yóò gbé ibùdó ìgbókùsíí kúrò nílé ìwòsàn."

Stage 1 Benchmark Sample 2: "Ṣèyí ń ṣe bíi Baṣọ̀run Gáà, tí kò bọ̀wọ̀ fún òfin."

Stage 1 Benchmark Sample 3: "Makinde ní kò sí ọwọ́jà ààrùn àìmọ̀ ẹlẹ́ẹ̀kejì ní ìpínlẹ̀ Ọ̀yọ́."

Repository Files

This repository contains all files necessary to run standalone synthesis or merge with standard Coqui pipelines:

How to Use

Ensure coqui-tts and dependencies are installed:

pip install coqui-tts "transformers<4.45.0" torch torchaudio huggingface_hub

Load the model and synthesize speech directly from this Hugging Face repository:

import os
import re
import torch
import torchaudio
from huggingface_hub import snapshot_download
from TTS.tts.configs.xtts_config import XttsConfig
from TTS.tts.models.xtts import Xtts

# 1. Download model repository
model_dir = snapshot_download(repo_id="samuelolubukun/xtts-v2-yoruba-openbible-conversational")

# 2. Load model config and checkpoint
config = XttsConfig()
config.load_json(os.path.join(model_dir, "config.json"))
model = Xtts.init_from_config(config)
model.load_checkpoint(
    config,
    checkpoint_path=os.path.join(model_dir, "model.pth"),
    vocab_path=os.path.join(model_dir, "vocab.json"),
    use_deepspeed=False
)
model.cuda() if torch.cuda.is_available() else model.cpu()

# 3. Text cleaner fallback for custom language support
orig_preprocess = model.tokenizer.preprocess_text
def safe_preprocess_text(txt, lang):
    try:
        return orig_preprocess(txt, lang)
    except NotImplementedError:
        txt = txt.lower()
        return re.sub(r"\s+", " ", txt).strip()
model.tokenizer.preprocess_text = safe_preprocess_text

# 4. Compute speaker conditioning latents (using bundled reference audio or your own 3-10s WAV)
ref_audio = os.path.join(model_dir, "samples/sample1_ref.wav")
gpt_cond_latent, speaker_embedding = model.get_conditioning_latents(
    audio_path=ref_audio,
    gpt_cond_len=30,
    max_ref_length=30,
    sound_norm_refs=False,
)

# 5. Synthesize speech
text = "Ẹ n lẹ́ o, báwo ni gbogbo yín ṣe wà? Èkó dára ju Àbújá lọ, àbí bẹ́ẹ̀ kọ́?"
out = model.inference(
    text=text,
    language="yo",
    gpt_cond_latent=gpt_cond_latent,
    speaker_embedding=speaker_embedding,
    temperature=0.3,
    length_penalty=1.0,
    repetition_penalty=5.0,
    top_k=30,
    top_p=0.85,
)

# 6. Save output audio (24,000 Hz)
torchaudio.save("output_yo.wav", torch.tensor(out["wav"]).unsqueeze(0), 24000)
print("Saved synthesized speech to output_yo.wav")

Limitations and Ethical Usage

• Conversational Acoustics vs. Studio Purity: While Stage 2 completely cures the sluggish scripture cadence and restores authentic tonal glides, Common Voice recordings contain diverse crowd-sourced microphones and acoustic environments. Reference audios with heavy background chatter or clipping should be avoided.
• Tone Marking Sensitivity: Yoruba is a tonal language where pitch alters meaning (ọkọ, ọkọ̀, ọkọ́). Always provide proper tone marks (acute ´, grave `) and sub-dots (ẹ, ọ, ṣ) in input text for optimal phonetics.
• Ethical Voice Cloning: This model is intended for research, linguistic preservation, and accessibility in African languages. Explicit consent should be obtained prior to cloning recognizable individual voices.
• Licensing: Inherits the Coqui Public Model License / MRCUL.

Author & Attribution

Developed and fine-tuned by Samuel Olubukun:

• Website: samuelolubukun.com
• GitHub: @samolubukun
• Hugging Face: @samuelolubukun

Citation

If you use this model or checkpoints in your research, applications, or benchmarks, please cite:

@misc{olubukun2026xttsyorubaconversational,
  author = {Samuel Olubukun},
  title = {XTTS-v2 Yoruba Conversational: 5-Epoch Domain Adaptation on Nigerian Common Voice},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/samuelolubukun/xtts-v2-yoruba-conversational}}
}

Acknowledgments

• Base Model: Coqui XTTS-v2
• Foundation Model: samuelolubukun/xtts-v2-yoruba-openbible
• Dataset: Benjamin Ogbonna for curating and releasing benjaminogbonna/nigerian_common_voice_dataset, and the Mozilla Common Voice contributors.