YuE2-3B Diffusers
A 69-second song generated with the code below, seed 831001.
Diffusers version of m-a-p/YuE2-3B and m-a-p/YuE2-Vae: the transformer, the VAE and the tokenizer. Run it with the blocks in OzzyGT/YuE2-Modular.
Note: This model requires tiktoken and the example uses soundfile, install them with pip install tiktoken soundfile.
Licensing
The weights are CC BY-NC 4.0 (non-commercial), from m-a-p; see WEIGHTS_LICENSE. The code is Apache-2.0, adapted from multimodal-art-projection/YuE; see LICENSE.
Sample song
The song above was generated with the following code:
import soundfile as sf
import torch
from diffusers import ModularPipelineBlocks
blocks = ModularPipelineBlocks.from_pretrained( # load the blocks first to avoid warnings
"OzzyGT/YuE2-Modular",
trust_remote_code=True,
components_repo="OzzyGT/YuE2-3B-Diffusers",
trust_components_code=True,
)
pipe = blocks.init_pipeline()
pipe.load_components(dtype={"transformer": torch.bfloat16, "vae": torch.float32})
pipe.to("cuda")
style = "English, warm piano pop, expressive female voice, acoustic piano, rounded bass and light drums, lyrical memorable melody, unhurried phrasing, 88 BPM"
lyrics = """[Verse]
Neon fades along the lane
Footsteps keep the time of rain
Fold the night and leave it here
Morning has a sky to clear
[Chorus]
Let the day come into view
Every road begins with you
Hold a little room for light
We will sing beyond the night"""
state = pipe(style=style, lyrics=lyrics, seed=831001, use_cuda_graph=True)
sf.write("song.flac", state.get("audios")[0].T.numpy(), state.get("sample_rate"), subtype="PCM_24")