Magpie-TTS Saudi Voice Clone

Zero-shot voice-cloning text-to-speech for Saudi Arabic, with Modern Standard Arabic and English retained from the base model. Fine-tuned by IST Networks from NVIDIA's magpie_tts_multilingual_357m.

Give it a 3 to 10 second clip of any speaker and a sentence; it speaks the sentence in that voice. There are no built-in speakers: the reference clip is required for every request.

Listen to samples/ before integrating. ar-SA_*.wav are Saudi sentences, en_*.wav are one English sentence spoken through Saudi and English voices.

Contents

magpie_tts_saudi_step78000.nemo        fine-tuned model (1.5 GB)
codec/nemo-nano-codec-22khz-*.nemo     audio codec required at inference (425 MB)
inference.py                           CLI + Python API, single sentence or batch
voices/                                reference clips: male_saudi, female_saudi, adnan, aria_en, sofia_en
samples/                               outputs from this checkpoint
training/                              the fine-tuning recipe (config, launcher, data scripts, logs)

Quick start

Environment

Tested with Python 3.11, PyTorch 2.11 (CUDA 12.9), NVIDIA RTX A6000. The Magpie-TTS inference modules live in the NeMo Speech repository, pinned to the commit the model was trained with.

git clone https://github.com/NVIDIA-NeMo/Speech.git
cd Speech && git checkout ca3f93a516ff172e32951928aea6f96a977a7600
pip install -e ".[tts]"
pip install soundfile huggingface_hub

Alternatively, the NVIDIA NeMo container nvcr.io/nvidia/nemo:25.xx with the repository above checked out on top works the same way.

Download

hf download ISTNetworks/magpie --local-dir magpie-tts-saudi
cd magpie-tts-saudi

Speak one sentence

python inference.py \
  --text "هلا والله، كيف الحال؟ اليوم الجو حلو مرة." \
  --ref voices/male_saudi.wav --lang ar-SA --out hello.wav

English in the same voice:

python inference.py --text "Good morning, and thank you for joining us." \
  --ref voices/male_saudi.wav --lang en --out hello_en.wav

Batch

One request per line in a JSONL file:

{"text": "أبشر، بكرة أرسل لك كل التفاصيل.", "context_audio_filepath": "voices/female_saudi.wav", "language": "ar-SA", "name": "greeting_01"}
{"text": "Your order has shipped.", "context_audio_filepath": "voices/adnan.wav", "language": "en", "name": "order_en"}

python inference.py --manifest requests.jsonl --out_dir outputs/

Python API (for a service)

from inference import MagpieTTS

tts = MagpieTTS("magpie_tts_saudi_step78000.nemo",
                "codec/nemo-nano-codec-22khz-1.89kbps-21.5fps.nemo")   # load once, ~40 s

wav, sr = tts.synthesize("النص هنا", ref_wav="voices/female_saudi.wav", lang="ar-SA")
# wav: float32 numpy array, sr: 22050

synthesize_batch([...]) takes a list of Request(text, ref_wav, lang) and runs them in one GPU pass. Requests with different languages are grouped automatically.

Languages

Dialect comes from the reference voice, not from the language code. ar-SA and ar use the same character set; the distinction exists so you can pass MSA text with diacritics when you have them.

Reference clip rules

• 3 to 10 seconds. Shorter clips are rejected; longer ones are cut to the first 10 seconds.
• One speaker, clean speech, no music or crosstalk. 6 to 9 seconds of natural sentences works best.
• Mono 22.05 kHz WAV is what training used. Other rates are resampled on load.
• The clip's language does not need to match the text. Saudi reference with English text works, see samples/en_*.wav.

Sampling parameters

Defaults match the released samples: temperature 0.6, top-k 80, classifier-free guidance scale 2.5. Lower temperature (0.4 to 0.5) is more stable for long production text; --no_cfg roughly halves latency at a quality cost.

Performance on one RTX A6000: model load about 40 s; a batch of 3 sentences of 5 to 10 s each synthesizes in 3 to 7 s after warm-up, roughly real-time factor 0.3.

Known limitations

• Digits and symbols are dropped in Arabic. The Arabic character tokenizer has no entries for 0-9, ٠-٩, %, or the tatweel ـ. A sentence containing "30 ألف" is spoken as "ألف". Convert numbers to words before synthesis. English text is unaffected; the phoneme tokenizer normalizes digits.
• Long paragraphs are split into sentences internally. Prosody across sentence boundaries is not modeled, so for voice-over work, synthesize sentence by sentence and concatenate.
• Voice similarity for speakers far from the training distribution (children, heavily processed audio) has not been evaluated.
• MagpieTTSModel.do_tts() from the base model does not work with this checkpoint. It relies on the baked speaker table that was removed. Use inference.py or the manifest-driven examples/tts/magpietts_inference.py.

Building a production container

Sketch of what a Dockerfile needs, in order:

1. Base image with CUDA 12.x and Python 3.11, or nvcr.io/nvidia/nemo.
2. Clone the NeMo Speech repository at commit ca3f93a5 and pip install -e ".[tts]".
3. Copy this repository's *.nemo, codec/, voices/, and inference.py into the image, or mount them as a volume.
4. Set TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 in the environment. inference.py also sets it, but a server entrypoint that imports NeMo before inference.py needs it earlier.
5. Wrap MagpieTTS in your HTTP layer. Load the model once at startup; the class is not thread-safe, so serialize calls or run one worker per GPU.

The .nemo archive contains the model config and weights; the codec is loaded separately, which is why inference.py takes both paths.

Fine-tuning recipe

Everything needed to reproduce or continue training is in training/.

Starting point

nvidia/magpie_tts_multilingual_357m (v2607) ships five baked-in English voices and, because of that table, never trains its context encoder. Two preparation steps unlock cloning:

1. strip_baked.py removes the baked speaker table from the checkpoint. With it gone, the model uses reference audio for speaker identity.
2. inject_ctx_encoder.py writes freshly initialized weights for the 18 M-parameter context encoder, whose tensors the loader requires.

Data

build_manifests.py pairs each utterance with a same-speaker reference clip (3 to 10 s), holds out 5 male and 3 female speakers entirely for cloning evaluation, and creates the undiacritized ClArTTS twin so the model learns both surface forms. Manifest lines carry context_audio_filepath and never context_text; a populated context_text would silently replace the reference audio and the encoder would not train. manifest_example.jsonl shows the shape.

Training

train_launcher.py wraps the stock examples/tts/magpietts.py flow with two additions: a separate learning rate for the context encoder, and a gradient probe that logs the encoder's gradient norm so a silent "encoder never trains" failure is caught on step 0.

Run order: run_smoke.sh (3 steps, asserts nonzero encoder gradient) then gate_and_train.sh, which starts run_train.sh only if the smoke test passed. gate.log records the gate decision for this run.

Validation loss

Full history in val_loss_history.csv; the complete training log is train.log.gz.

Train loss at the end was 9.7, a 0.5 gap to validation, so mild overfitting had begun but was not yet audible. The curve was flat from epoch 33 onward; a longer run at this learning rate would not help. More Saudi data, or number verbalization in the text, is the next lever.

Export

export_nemo.py loads a Lightning .ckpt with its hparams.yaml and saves a standalone .nemo. hparams_step78000.yaml is the config the released checkpoint was trained with.

License

The base model and codec are released by NVIDIA under the NVIDIA Open Model License. This fine-tune inherits those terms. The Saudi speech data used for fine-tuning is proprietary to IST Networks and is not included; the five clips in voices/ are provided only as reference-voice examples for inference.