Fun-CosyVoice3-0.5B-2512 — GGUF

GGUF conversion of FunAudioLLM/Fun-CosyVoice3-0.5B-2512 for use with CrispASR (--backend cosyvoice3-tts).

CosyVoice3 is a streaming, multilingual, zero-shot voice-cloning TTS system from Alibaba's FunAudioLLM team. The 0.5B-2512 release is Apache-2.0 licensed and supports 9 languages plus 18 Chinese dialects. Output is 24 kHz mono.

The model is a three-stage pipeline:

text (Qwen2 BPE) → CosyVoice3LM (Qwen2-0.5B + speech-token AR head)
                 → speech tokens ∈ [0, 6561)
                 → Flow (DiT + CausalConditionalCFM, 10-step Euler ODE)
                 → mel @ 24 kHz / 480-hop
                 → CausalHiFTGenerator (HiFi-GAN + NSF + iSTFT)
                 → 24 kHz PCM

Files

Base vs RL talker

Upstream ships two talker checkpoints in one repo: llm.pt (pre-trained) and llm.rl.pt (the same architecture after reinforcement learning, tuned by the authors for speech quality, pronunciation accuracy and generation stability). Only the talker differs — flow, HiFT, CAMPPlus, the speech tokenizer and the voice bank are shared, so switching is just pointing -m at the other LLM GGUF and leaving every companion file in place:

crispasr -m cosyvoice3-llm-rl-q4_k.gguf --backend cosyvoice3-tts \
    --voice fleurs-en --i-have-rights \
    --tts "The northern lights can be heard as well as seen." --tts-output out.wav

CrispASR also ships --backend cosyvoice3-tts-rl (aliases cosyvoice3-rl, cv3-rl): the same engine, wired so that -m auto fetches the RL talker and reuses the shared companions.

crispasr --backend cosyvoice3-tts-rl -m auto \
    --voice fleurs-en --i-have-rights \
    --tts "The northern lights can be heard as well as seen." --tts-output out.wav

Both talkers are published here; keep whichever you prefer and delete the other.

Pick one LLM + one flow + HiFT + voices. The smallest viable combo is llm-q4_k + flow-q8_0 + hift-f16 + voices at 745 MB total; the F16 reference is 1.96 GB. The s3tok + campplus companions are only needed for arbitrary-WAV runtime cloning (below) — not for synthesis with a baked voice.

Quant validation (ASR roundtrip on smoke prompt)

Synthesis used the default zero-shot voice (upstream asset/zero_shot_prompt.wav) at --temperature 0.8 --seed 42. The generated WAV was transcribed with parakeet-tdt-0.6b-v3-q4_k and compared against the prompt text.

Q4_K LLM introduces a small punctuation drift (commas occasionally read as question-intonation) but content is fully preserved across languages. Q8_0 flow is perceptually indistinguishable from F16.

Usage

CrispASR (recommended)

# Auto-discovers flow + hift + voices as siblings of the LLM.
crispasr -m cosyvoice3-llm-q4_k.gguf \
         --backend cosyvoice3-tts \
         --tts "Hello, this is a test." \
         --voice zero_shot \
         --tts-output out.wav

The CLI auto-discovers companion GGUFs in this order:

• Flow — cosyvoice3-flow-*.gguf next to the LLM, or --codec-model PATH.
• CAMPPlus — cosyvoice3-campplus-f16.gguf next to the LLM, or COSYVOICE3_CAMPPLUS_PATH for the native WAV-clone path.
• S3Tokenizer — cosyvoice3-s3tok-f16.gguf next to the LLM, or COSYVOICE3_S3TOK_PATH for the native WAV-clone path.
• HiFT — cosyvoice3-hift-*.gguf next to the LLM, or COSYVOICE3_HIFT_PATH env var.
• Voices — cosyvoice3-voices.gguf next to the LLM, or COSYVOICE3_VOICES_PATH env var.

Greedy decode is disabled by default (CV3 falls into a documented "silent_tokens" loop within ~5 steps). The backend overrides --temperature 0 to 0.8 so the RAS sampler engages — pass a different positive value to override.

Voices

cosyvoice3-voices.gguf ships a small multilingual voice bank — pass the name to --voice:

The fleurs-* prompts are ~4–6 s clips from Google's FLEURS corpus (CC BY 4.0), loudness-normalised before baking. CV3 clones the prompt's timbre and level, so quiet prompts yield quiet output — normalise your own prompt clips for a consistent level. More voices can be baked with the converter in the CrispASR tree:

python models/convert-cosyvoice3-voices-to-gguf.py \
    --manifest my-voices.json \
    --upstream-base /path/to/CosyVoice-clone \
    --output my-voices.gguf \
    --i-have-rights

Each manifest entry is {name, wav, prompt_text}. The script needs campplus.onnx (CV2/CV3 speaker encoder) and speech_tokenizer_v3.onnx (CV3 token extractor); both auto-download from HF on first run.

--i-have-rights is required: baking turns a real speaker's recording into a reusable voice clone, and by passing it you attest that you have that speaker's consent or that the voice is your own. The converter stamps each entry as recording-derived, which is what makes CrispASR demand --i-have-rights / consent_attestation at synthesis time and prepend the spoken AI disclosure to the output (EU AI Act Art. 50(4)).

A bundle baked before that stamp existed still gates — every entry falls back to the producer architecture — but re-bake it for per-entry accuracy.

Arbitrary-WAV cloning (native, no Python pre-bake)

With the cosyvoice3-s3tok-f16.gguf + cosyvoice3-campplus-f16.gguf companions present (siblings of the LLM, or pulled by -m auto), you can clone from any 16 kHz WAV at runtime:

crispasr -m cosyvoice3-llm-q4_k.gguf \
         --backend cosyvoice3-tts \
         --voice my_reference.wav \
         --ref-text "exact transcription of my_reference.wav" \
         --tts "The text to speak in the cloned voice." \
         --tts-output out.wav

The runtime ports all three front-end extractors to ggml: the speech_tokenizer_v3 token extractor (12 FSMN/attention blocks + FSQ head — byte-exact vs the ONNX reference, validated stage-by-stage with crispasr-diff), the CAMPPlus 192-D speaker encoder, and the matcha 24 kHz reference mel. The legacy Python pre-bake bridge (convert-cosyvoice3-voices-to-gguf.py) remains as an automatic fallback when the companions are absent.

Tensor naming

Conventional naming for all three GGUFs:

• LLM — llama.cpp-standard token_embd, blk.K.{attn,ffn}_*, output_norm, output, plus CV3-specific cosyvoice3.speech_embd.weight (input embedding, vocab 6761) and cosyvoice3.speech_lm_head.weight (output head).
• Flow — cosyvoice3.flow.{input_embd,pre_la,spk_affine,dit.*} matching the upstream CausalMaskedDiffWithDiT module tree.
• HiFT — cosyvoice3.hift.{conv_pre,ups.K,resblocks.K.*,source_*, m_source,f0.*,conv_post} with weight-norm pre-resolved on the Python converter side (g · v / ‖v‖).

License

The model weights are Apache-2.0 (inherited from the upstream model). Free for commercial use. The zero_shot voice prompt is the asset/zero_shot_prompt.wav clip from the Apache-2.0 CosyVoice repo.

The fleurs-{en,de,zh,ja,fr,es,ko} voice prompts are derived (trimmed + loudness-normalised) from Google's FLEURS corpus, licensed CC BY 4.0 — commercial use permitted, attribution required:

FLEURS (Few-shot Learning Evaluation of Universal Representations of Speech), Conneau et al., 2022 — https://huggingface.co/datasets/google/fleurs, licensed CC BY 4.0. The prompt clips here are trimmed excerpts, loudness-normalised; no other modification.

All eight baked voices are therefore clean for commercial use under permissive licenses (Apache-2.0 / CC BY 4.0).

Related links

• Upstream: FunAudioLLM/Fun-CosyVoice3-0.5B-2512
• Project page: funaudiollm.github.io/cosyvoice3
• Code: github.com/FunAudioLLM/CosyVoice
• CrispASR: github.com/CrispStrobe/CrispASR