Vyvo-Multilingual-JA-FT-v0.1
A Japanese voice-cloning TTS model — Vyvo/Vyvo-Multilingual-v0.1 fine-tuned for a single conversational Japanese speaker. Give a short reference clip (+ its transcript) and a target text; the model speaks the text in that voice by pure language-model continuation.
- Backbone: Qwen3-0.6B + kyutai/mimi audio codec (32 codebooks, 24 kHz)
- Base model: Vyvo/Vyvo-Multilingual-v0.1
- Goal: improve Japanese intelligibility for this voice over the multilingual base
Results
Japanese Character Error Rate (CER — the standard metric for Japanese) on 100 evaluation texts, transcribed with Qwen3-ASR-1.7B (lower = better):
| Model | Decoding | CER ↓ | WER ↓ | Accuracy (1-CER) |
|---|---|---|---|---|
| Base (Vyvo-Multilingual-v0.1) | greedy | 0.395 | 0.426 | 60.5 % |
| This model | greedy | 0.338 | 0.375 | 66.2 % |
| This model | sampling + best-of-6 | 0.155 | 0.218 | 84.5 % |
Greedy decoding under-reports quality (occasional early end-of-speech); with sampling + best-of-N the model reaches ~84.5 % character accuracy. Most remaining errors are homophone / kanji-vs-kana substitutions that are phonetically correct.
Recommended decoding
The model is a transcript-conditioned TTS: always pass the reference clip's
transcript. For production use sampling (temperature ≈ 0.4–0.5, top_p = 0.9,
repetition_penalty = 1.1) and, when quality matters, best-of-N (generate
4–6 candidates and keep the one whose ASR transcript best matches the target).
Usage
import soundfile as sf
import torch
import torchaudio
from transformers import (AutoModelForCausalLM, AutoTokenizer,
AutoFeatureExtractor, MimiModel)
REPO = "Vyvo/Vyvo-Multilingual-JA-FT-v0.1"
DEVICE = "cuda"
BASE = 151669
NUM_CODEBOOKS, CODEBOOK_SIZE, AUDIO_OFFSET = 32, 2048, 10
SOS, EOS, SOH, EOH, SOA = 1, 2, 3, 4, 5
tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16).to(DEVICE).eval()
mimi = MimiModel.from_pretrained("kyutai/mimi").to(DEVICE).eval()
feature_extractor = AutoFeatureExtractor.from_pretrained("kyutai/mimi")
def encode_reference(path):
wav, sr = sf.read(path, dtype="float32", always_2d=False)
if wav.ndim > 1:
wav = wav.mean(axis=1)
target_sr = feature_extractor.sampling_rate
if sr != target_sr:
wav = torchaudio.functional.resample(
torch.from_numpy(wav).unsqueeze(0), sr, target_sr).squeeze(0).numpy()
inputs = feature_extractor(raw_audio=wav, sampling_rate=target_sr, return_tensors="pt")
codes = mimi.encode(inputs["input_values"].to(DEVICE),
num_quantizers=NUM_CODEBOOKS).audio_codes[0].cpu()
flat = codes.transpose(0, 1).reshape(-1).tolist()
return [c + AUDIO_OFFSET + (i % NUM_CODEBOOKS) * CODEBOOK_SIZE + BASE
for i, c in enumerate(flat)]
def build_prompt(ref_tokens, ref_text, target_text):
text_ids = tokenizer(ref_text + " " + target_text, add_special_tokens=False).input_ids
head = [BASE + SOH] + text_ids + [tokenizer.eos_token_id, BASE + EOH, BASE + SOA, BASE + SOS]
return head + ref_tokens
def decode_audio(generated_ids):
codes = []
for i, tok in enumerate(generated_ids):
v = tok - BASE - AUDIO_OFFSET - (i % NUM_CODEBOOKS) * CODEBOOK_SIZE
if 0 <= v < CODEBOOK_SIZE:
codes.append(v)
else:
break
frames = len(codes) // NUM_CODEBOOKS
codes = torch.tensor(codes[:frames * NUM_CODEBOOKS]).view(frames, NUM_CODEBOOKS)
codes = codes.t().unsqueeze(0).to(DEVICE)
return mimi.decode(codes).audio_values.squeeze().cpu().float().numpy()
reference_wav = "reference.wav"
reference_text = "リファレンス音声の書き起こし" # transcript of the reference clip
target_text = "こんにちは、これは音声合成モデルのテストです。"
prompt = build_prompt(encode_reference(reference_wav), reference_text, target_text)
input_ids = torch.tensor([prompt], device=DEVICE)
output = model.generate(
input_ids, attention_mask=torch.ones_like(input_ids),
max_new_tokens=9600, min_new_tokens=960,
do_sample=True, temperature=0.45, top_p=0.9, repetition_penalty=1.1,
eos_token_id=BASE + EOS, pad_token_id=tokenizer.eos_token_id)
audio = decode_audio(output[0, input_ids.shape[1]:].tolist())
sf.write("output.wav", audio, mimi.config.sampling_rate)
Training
| Method | full fine-tuning (liger fused cross-entropy) |
| Learning rate | 1e-4, cosine, warmup 0.03 |
| Epochs / batch | 2 / 8 |
| Hardware | 1× H100 |
Recipe carried over from the best English fine-tune (full FT, lr 1e-4).
Limitations
- Requires the reference transcript. Transcript-free / zero-shot cloning collapses.
- Single speaker / Japanese only. CER over-penalises homophone kanji/kana variants, so true intelligibility is somewhat higher than the number suggests.
- Greedy decoding may truncate long sentences — see Recommended decoding.
License
Fine-tuned from Vyvo/Vyvo-Multilingual-v0.1. Released under the MIT License.