Vyvo-Multilingual-EN-FT-v0.1
An English voice-cloning TTS model — Vyvo/Vyvo-Multilingual-v0.1 fine-tuned for a single expressive English speaker. Give a short reference clip (+ its transcript) and a target text; the model speaks the text in that voice by pure language-model continuation.
- Backbone: Qwen3-0.6B + kyutai/mimi audio codec (32 codebooks, 24 kHz)
- Base model: Vyvo/Vyvo-Multilingual-v0.1
- Goal: sharply improve English intelligibility / WER for this voice over the multilingual base
Results
English Word Error Rate on 100 evaluation texts, transcribed with Qwen3-ASR-1.7B (lower = better):
| Model | Decoding | WER ↓ | Accuracy |
|---|---|---|---|
| Base (Vyvo-Multilingual-v0.1) | greedy | 0.485 | 51.5 % |
| This model | greedy | 0.248 | 75.2 % |
| This model | sampling + best-of-6 | 0.067 | 93.3 % |
Fine-tuning roughly halves the word error rate. Greedy decoding under-reports quality because of occasional early end-of-speech; with sampling + best-of-N the model reaches ~93 % word accuracy.
Recommended decoding
The model is a transcript-conditioned TTS: always pass the reference clip's
transcript. Greedy can truncate long sentences, so for production use sampling
(temperature ≈ 0.4–0.5, top_p = 0.9, repetition_penalty = 1.1) and, when
quality matters, best-of-N (generate 4–6 candidates and keep the one whose ASR
transcript best matches the target).
Usage
import soundfile as sf
import torch
import torchaudio
from transformers import (AutoModelForCausalLM, AutoTokenizer,
AutoFeatureExtractor, MimiModel)
REPO = "Vyvo/Vyvo-Multilingual-EN-FT-v0.1"
DEVICE = "cuda"
# Token layout (must match training)
BASE = 151669
NUM_CODEBOOKS, CODEBOOK_SIZE, AUDIO_OFFSET = 32, 2048, 10
SOS, EOS, SOH, EOH, SOA = 1, 2, 3, 4, 5 # special tokens, as offsets above BASE
tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16).to(DEVICE).eval()
mimi = MimiModel.from_pretrained("kyutai/mimi").to(DEVICE).eval()
feature_extractor = AutoFeatureExtractor.from_pretrained("kyutai/mimi")
def encode_reference(path):
wav, sr = sf.read(path, dtype="float32", always_2d=False)
if wav.ndim > 1:
wav = wav.mean(axis=1)
target_sr = feature_extractor.sampling_rate
if sr != target_sr:
wav = torchaudio.functional.resample(
torch.from_numpy(wav).unsqueeze(0), sr, target_sr).squeeze(0).numpy()
inputs = feature_extractor(raw_audio=wav, sampling_rate=target_sr, return_tensors="pt")
codes = mimi.encode(inputs["input_values"].to(DEVICE),
num_quantizers=NUM_CODEBOOKS).audio_codes[0].cpu()
flat = codes.transpose(0, 1).reshape(-1).tolist()
return [c + AUDIO_OFFSET + (i % NUM_CODEBOOKS) * CODEBOOK_SIZE + BASE
for i, c in enumerate(flat)]
def build_prompt(ref_tokens, ref_text, target_text):
text_ids = tokenizer(ref_text + " " + target_text, add_special_tokens=False).input_ids
head = [BASE + SOH] + text_ids + [tokenizer.eos_token_id, BASE + EOH, BASE + SOA, BASE + SOS]
return head + ref_tokens
def decode_audio(generated_ids):
codes = []
for i, tok in enumerate(generated_ids):
v = tok - BASE - AUDIO_OFFSET - (i % NUM_CODEBOOKS) * CODEBOOK_SIZE
if 0 <= v < CODEBOOK_SIZE:
codes.append(v)
else:
break
frames = len(codes) // NUM_CODEBOOKS
codes = torch.tensor(codes[:frames * NUM_CODEBOOKS]).view(frames, NUM_CODEBOOKS)
codes = codes.t().unsqueeze(0).to(DEVICE)
return mimi.decode(codes).audio_values.squeeze().cpu().float().numpy()
reference_wav = "reference.wav"
reference_text = "text spoken in the reference clip"
target_text = "Hello, this is a test of the text to speech model."
prompt = build_prompt(encode_reference(reference_wav), reference_text, target_text)
input_ids = torch.tensor([prompt], device=DEVICE)
output = model.generate(
input_ids, attention_mask=torch.ones_like(input_ids),
max_new_tokens=9600, min_new_tokens=960,
do_sample=True, temperature=0.45, top_p=0.9, repetition_penalty=1.1,
eos_token_id=BASE + EOS, pad_token_id=tokenizer.eos_token_id)
audio = decode_audio(output[0, input_ids.shape[1]:].tolist())
sf.write("output.wav", audio, mimi.config.sampling_rate)
Training
Full fine-tune of every weight, continued from the base checkpoint (vocabulary already resized for the Mimi tokens).
| Method | full fine-tuning (liger fused cross-entropy) |
| Learning rate | 1e-4, cosine, warmup 0.03 |
| Epochs / batch | 4 / 8 |
| Hardware | 1× H100 |
A learning-rate sweep (1e-5 → 5e-5 → 1e-4) and a LoRA run were compared; full fine-tuning at 1e-4 won (LoRA reached WER 0.329, lr 5e-5 reached 0.310).
Limitations
- Requires the reference transcript. Transcript-free / zero-shot cloning collapses (the model speaks short, unrelated fragments).
- Single speaker / English only. Rare proper nouns are occasionally mis-rendered.
- Greedy decoding may truncate long sentences — see Recommended decoding.
License
Fine-tuned from Vyvo/Vyvo-Multilingual-v0.1. Released under the MIT License.