Qwen3-TTS-12Hz-1.7B-Vi

Vietnamese text-to-speech with zero-shot voice cloning, finetuned from Qwen/Qwen3-TTS-12Hz-1.7B-Base.

The base model covers 10 languages and Vietnamese is not one of them — it produces the right shape of Vietnamese with wrong tones and wrong vowels. This model drops Vietnamese CER from 0.253 to 0.065 while speaker similarity falls only 0.006, so cloning still works on voices it has never heard.

Clone any voice from a few seconds of reference audio. English is unaffected.

Listen

Same reference voice, same sentence, base model vs this one. Everything below is zero-shot cloning — no speaker is baked in.

Reference voice (the target timbre):

What the base model does to sentence 3, transcribed back by Whisper:

ref: Hóa đơn của bạn đã được xuất thành công, tổng tiền là hai triệu ba trăm nghìn đồng.
hyp: Họ đơn quà bạn đã được sử dụng, đồng đến là hại chú 8 chẳng mến đồng.   <- base

A second speaker

Cloning a voice that was never in training

Measured on the x-vector path only. These clips pass the reference audio without a transcript. Cloning an unseen voice through the ref_text path has not been evaluated — see Evaluation coverage.

Reference:

Speaker cosine against that reference: 0.9609.

English is not damaged

Every generated clip is under samples/.

Usage

Identical to the base model — tts_model_type is still "base", so create_voice_clone_prompt and generate_voice_clone work unchanged.

import soundfile as sf, librosa, torch
from qwen_tts import Qwen3TTSModel

tts = Qwen3TTSModel.from_pretrained(
    "beyoru/Qwen3-TTS-12Hz-1.7B-Vi",
    device_map="cuda:0", dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

ref_wav, ref_sr = librosa.load("reference.wav", sr=None, mono=True)

# Pass ref_text when you have a transcript. This is what generate_voice_clone
# defaults to, and the path this finetune was built to repair -- but see
# "Evaluation coverage": it was only measured on training voices.
prompt = tts.create_voice_clone_prompt(
    ref_audio=(ref_wav, ref_sr),
    ref_text="transcript of reference.wav",
)

wavs, sr = tts.generate_voice_clone(
    text="Xin chào, tôi là trợ lý ảo, rất vui được gặp bạn.",
    language="Auto",                 # keep Auto -- see note below
    voice_clone_prompt=prompt,
    do_sample=True, temperature=0.9, top_k=50, repetition_penalty=1.05,
)
sf.write("out.wav", wavs[0], sr)

Without a transcript, use the x-vector path:

prompt = tts.create_voice_clone_prompt(
    ref_audio=(ref_wav, ref_sr), ref_text=None, x_vector_only_mode=True,
)

Keep language="Auto". codec_language_id has no Vietnamese entry (only zh/en/de/it/pt/es/ja/ko/fr/ru). Auto routes through codec_nothink_id, which is what this model expects.

Spell foreign-looking proper nouns with a space or hyphen — Misa → Mi-sa. See Limitations.

Results

CER from Whisper-large-v3-turbo transcribing the generated audio back, over 24 sentences covering both cloning paths and both languages. All 24 use voices that were in training — see Evaluation coverage for what that does and does not establish.

Speaker cosine, using the model's own speaker encoder:

Limitations

Evaluation coverage

What was actually measured, so you can judge what transfers to your use case:

Every CER and cosine number above comes from the two training voices. The held-out voice was only run through the x-vector path (cosine 0.9609).

This matters because the ref_text path is what generate_voice_clone uses by default whenever you pass a transcript, and it is the path this finetune was specifically built to repair. Cloning an unseen voice through that path is untested here. If that is your use case, evaluate it before relying on it.

The cause was mundane: the generation script only enables the ref_text path when a reference transcript is supplied, and none was supplied for the held-out clip, so it silently fell back to x-vector only.

Write foreign-looking names with a space or hyphen

The one real pronunciation defect, and it is fixable in your text front-end without touching the model.

Written Misa, the brand name comes out /miza/ roughly half the time — the intervocalic ⟨s⟩ gets voiced. The base model covers en/de/pt/es/it, all of which voice intervocalic ⟨s⟩ (music, Rose, casa), and a string that does not look Vietnamese gets read with that rule.

Same sentence, same reference voice, N generations per spelling:

Keep a pronunciation lexicon for brand names and proper nouns (Misa → Mi-sa), the way any production TTS front-end does. Two things not to do: lowering temperature is not a fix (50% wrong at T=0.9 is still 33% wrong at T=0.3 — the model is genuinely split between two readings, not sampling a rare tail), and adding diacritics backfires (Mít sa gets the /s/ right but geminates it to "Missa" in 6 of 6 runs).

Audio for every cell of that table is under samples/probe_misa/.

Northern accent, including its mergers

The training voices are Northern, so ⟨s⟩ and ⟨x⟩ are both realized [s]. se lạnh and xe lạnh are exact homophones in this voice and Whisper transcribes the former as the latter in every run — base model included. That is the accent working as intended, not a defect, but it means this is not a neutral or Southern voice.

Reading the CER

Two Whisper normalization artifacts inflate it and are not synthesis errors: mười tám is transcribed as 18, hai triệu ba trăm nghìn as 2 triệu 300 nghìn. Excluding the two sentences with numbers, CER is 0.046 on the ref_text path and 0.043 on the x-vector path.

Other

No Vietnamese language token (runs as Auto). Prosody on long or heavily-punctuated text is untested beyond the evaluation sentences.

Ethics

This model clones a voice from a few seconds of audio. Get consent from the person whose voice you are cloning. Do not use it to impersonate anyone, and label synthetic speech as synthetic.

License

apache-2.0, inherited from the base model.