UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
UNISON is a unified latent flow-matching framework for audio and speech generation and editing. Using a single set of weights, it integrates text-to-audio, text-to-speech, zero-shot speaker cloning, mixed speech-and-sound scene generation, and audio/speech-in-scene editing — all in one model, one architecture, one forward pass.
Model variants in this repository
This repository hosts two checkpoint variants:
| Directory | VAE | DiT depth | Channels | Config |
|---|---|---|---|---|
unison_D20S0_O_40ch/ | MMAudio 44 kHz | 20 double + 0 single | 40 | D20S0_O_40ch.yaml |
unison_D24S0_O_20ch/ | MMAudio 16 kHz | 24 double + 0 single | 20 | D24S0_O_20ch.yaml |
Both variants share the same Qwen2.5-Omni-7B text encoder and the same inference pipeline.
Supported tasks
| Task | Prompt format |
|---|---|
| Text-to-Audio (T2A) | [Audio] {caption} |
| Text-to-Speech (TTS) | [Speech] A {female/male} voice saying "{text}" |
| Mixed Speech + Sound | [Speech] A {gender} voice saying "{text}" [Audio] {background} |
| Zero-shot Speaker Cloning | [Speech with voice] {ref_text}, {target_text} |
| Audio Scene Editing (add / remove / replace / denoise) | [Edit] [Audio] {instruction} |
| Speech-in-Scene Editing (content / insert / delete) | [Edit] [Speech] {instruction} |
| Timed Temporal Composition | [Audio] From {t1}s to {t2}s, {event1}. From {t2}s to {t3}s, {event2}. ... |
Task identity is encoded via a mask channel; source/reference audio is injected through VAE-encoded channel concatenation — no separate encoders or task-specific heads needed.
Architecture
All tasks share the same VAE encoder/decoder, MM-DiT backbone, and forward pass. Text conditioning uses layer-wise deep LLM fusion: hidden states from uniformly sampled layers of the frozen Qwen2.5-Omni-7B backbone are injected into corresponding MM-DiT double-stream blocks via learned linear projections.
Quick start
1. Clone repo and install dependencies
git clone https://github.com/lizhaoqing/UNISON
cd UNISON
pip install -r requirements.txt
flash-attn is optional but strongly recommended (automatic fallback to PyTorch SDPA):
pip install flash-attn --no-build-isolation
2. MMAudio VAE weights
Download from the MMAudio release and place at:
unison/models/mmaudio/data/ext_weights/
v1-44.pth # 44 kHz VAE (for D20S0 / 44k variant)
v1-16.pth # 16 kHz VAE (for D24S0 / 16k variant)
best_netG.pt # BigVGAN vocoder (16 kHz VAE only)
3. Qwen2.5-Omni-7B
export QWEN_OMNI_MODEL_PATH=Qwen/Qwen2.5-Omni-7B
# or point to a local download:
export QWEN_OMNI_MODEL_PATH=/path/to/Qwen2.5-Omni-7B
4. Download checkpoints (this repo)
hf download jac22/UNISON --local-dir checkpoints
This produces:
checkpoints/
unison_D20S0_O_40ch/model.safetensors # 44 kHz
unison_D24S0_O_20ch/model.safetensors # 16 kHz
5. Run inference
cd UNISON
# 44 kHz variant (D20S0)
bash scripts/infer.sh \
--checkpoint_dir checkpoints/unison_D20S0_O_40ch \
--model_config unison/config/D20S0_O_40ch.yaml \
--vae_config unison/models/mmaudio/vae_config_44k.yaml \
--task_mode all
# 16 kHz variant (D24S0)
bash scripts/infer.sh \
--checkpoint_dir checkpoints/unison_D24S0_O_20ch \
--model_config unison/config/D24S0_O_20ch.yaml \
--vae_config unison/models/mmaudio/vae_config_16k.yaml \
--task_mode all
Outputs are written to <checkpoint_dir>/infer_<N>steps/<ckpt_name>/.
Single-prompt example
python unison/pipelines/infer.py \
--model_ckpt checkpoints/unison_D20S0_O_40ch \
--model_config unison/config/D20S0_O_40ch.yaml \
--vae_config unison/models/mmaudio/vae_config_44k.yaml \
--omni_model_path $QWEN_OMNI_MODEL_PATH \
--task_mode generation \
--gen_prompt "[Audio] Rain falling on a tin roof with distant thunder" \
--gen_duration 10.0 \
--output_dir outputs/demo
Key inference parameters
| Argument | Default | Description |
|---|---|---|
--num_inference_steps | 100 | ODE solver steps (50 for fast, 100 for paper quality) |
--guidance_scale | 4.5 | Classifier-free guidance scale |
--seed | 42 | Random seed |
--gen_duration | 10.0 | Output length in seconds (generation tasks) |
--ref_duration | 3.0 | Reference clip length in seconds (zero-shot TTS) |
Checkpoint format
Each checkpoint is a single model.safetensors file (unwrapped from EMA).
The inference pipeline also accepts:
- A directory — auto-detects
ema_model.pt→model.safetensors→pytorch_model.bin - A direct file path to any of the three formats
EMA wrappers are unwrapped automatically at load time.
License
This project is released under the Apache 2.0 License with additional non-commercial use restrictions inherited from upstream dependencies:
- The backbone architecture derives from HunyuanVideo (Tencent), which prohibits commercial use without a separate license.
- Text/audio conditioning uses Qwen2.5-Omni (Alibaba Cloud), subject to its own license terms.
This model is intended for research and non-commercial use only.
Citation
@article{li2026unison,
title = {UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion},
author = {Li, Zhaoqing and Xu, Haoning and Su, Jingran and Liu, Yaofang and Rao, Zhefan and
Wang, Huimeng and Deng, Jiajun and Wang, Tianzi and Jin, Zengrui and Liu, Rui and
Che, Haoxuan and Liu, Xunying},
journal = {arXiv preprint arXiv:2605.31530},
year = {2026}
}
Acknowledgements
We thank the authors of the following works for their excellent open-source contributions:
- HunyuanVideo — MM-DiT backbone architecture
- MMAudio — audio VAE and feature utilities
- Qwen2.5-Omni — text/audio LLM used for deep conditioning
- Ovi (Character.AI) — inspiring cross-modal fusion design for joint audio-video generation