LiveSynth
Weights for LiveSynth: A Streaming Neural Synthesizer for Instrument Cloning
and Text-to-Instrument (Kyungsu Kim, Yejin Kim, Kyogu Lee, Seoul National
University). Use them with the livesynth Python package, which downloads
this repository automatically:
from livesynth import LiveSynth
synth = LiveSynth.from_pretrained() # repo_id="KyungsuKim/LiveSynth"
audio = synth.render("song.mid", synth.embed_audio("reference.wav"))
| File | Contents |
|---|---|
generator.safetensors | 190 M-parameter feedback-free causal Transformer (Linear weights bf16, embeddings and norms fp32) |
decoder.safetensors | Causal Vocos latent decoder of the VAE codec (fp32) and latent standardisation statistics |
text_align.safetensors | Optional orthogonal Procrustes map (512 x 512) toward the audio embeddings the model was trained on (embed_text(..., align="procrustes"); the default uses the raw text embedding) |
presets.safetensors | CLAP timbre embeddings of 53 held-out NSynth instruments (names in config.json); each is the embedding of one reference recording in references/ |
references/*.flac | The 10-s reference recordings behind the presets (NSynth notes, 48 kHz), fetched on demand by synth.preset_audio(name) |
config.json | Model hyper-parameters and provenance |
The timbre encoder is the public LAION-CLAP checkpoint
lukewys/laion_clap/music_audioset_epoch_15_esc_90.14.pt, downloaded separately.
The weights are released under the MIT License, like the code.