spansynth-edit

Synthesise music from MIDI, or add, remove, and modify notes in a recording by revising its MIDI. SpanSynth-Edit generates the selected region, using surrounding audio for timbre guidance.

Try it in your browser: upload audio, transcribe with YourMT3+, edit the piano roll, and generate. See the local web setup to run the app yourself. You can also use the ComfyUI workflow and piano-roll editor.

Install

System requirements

• Python: 3.11–3.13 with PyTorch.
• NVIDIA GPU: CUDA with bfloat16 support. 6 GB VRAM recommended.
• Apple Silicon GPU: supported through Metal (PyTorch MPS). Tested on an M1 Pro with 16 GB unified memory and PyTorch 2.13.
• CPU inference is also supported.

Measured peak VRAM was about 3.9 GiB on a GH200 for synthesis and both editing methods with default settings (20.48 s crop, 16 steps, CFG 2.0).

Install PyTorch for your GPU first, then install the CLI without downloading the demo audio:

git clone --depth 1 --filter=blob:none --sparse https://github.com/mimbres/spansynth-edit.git
cd spansynth-edit
git sparse-checkout set spansynth
python -m pip install .

Model and codec weights download automatically on first use. No token is required.

Diffusers

Install the optional integration:

python -m pip install '.[diffusers]'

Download the example files, then edit the selected region:

import soundfile as sf
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    'mimbres/spansynth-edit',
    custom_pipeline='pipeline',
    trust_remote_code=True,
).to('cuda')
result = pipe(
    audio='../spansynth-inputs/early-slakh-track00006-original.mp3',
    midi='../spansynth-inputs/early-slakh-track00006-after.mid',
    num_inference_steps=16,
    guidance_scale=2.0,
)
sf.write('edited.wav', result.audios[0, 0], result.sample_rate, subtype='FLOAT')

Use .to('mps') for Apple Silicon or .to('cpu') for CPU. For FlowEdit, add method='flowedit' and source_midi pointing to the original score. The pipeline also accepts the crop, region, MIDI offset, and context options described below. This custom pipeline uses the installed spansynth package. Pass custom_pipeline='pipeline' to load the repository's pipeline.py implementation. Save and reopen the pipeline locally with:

pipe.save_pretrained('saved-pipeline')
pipe = DiffusionPipeline.from_pretrained(
    'saved-pipeline', custom_pipeline='pipeline', trust_remote_code=True,
)

Try an example

Defaults: spansynth-edit · 16 Euler steps · CFG 2.0 Audio context enabled · Context MIDI omitted

Example files

From the repository directory, download the sample audio and its original and revised MIDI files (about 0.5 MB total):

mkdir -p ../spansynth-inputs
for name in early-slakh-track00006-original.mp3 \
            early-slakh-track00006-before.mid \
            early-slakh-track00006-after.mid; do
  curl --fail --location --output "../spansynth-inputs/$name" \
    "https://raw.githubusercontent.com/mimbres/spansynth-edit/main/demo/assets/$name"
done

The examples below regenerate 6.40–14.08 s within the first 20.48 s of the recording.

spansynth-edit

Provide the full revised MIDI, including notes that should remain unchanged within the selected region:

spansynth-edit edit \
  --audio ../spansynth-inputs/early-slakh-track00006-original.mp3 \
  --midi ../spansynth-inputs/early-slakh-track00006-after.mid \
  --cfg 2.0 \
  --output ../spansynth-results/slakh-edit

spansynth-edit + flowedit

Provide both the original and revised MIDI:

spansynth-edit edit --method flowedit \
  --audio ../spansynth-inputs/early-slakh-track00006-original.mp3 \
  --source-midi ../spansynth-inputs/early-slakh-track00006-before.mid \
  --midi ../spansynth-inputs/early-slakh-track00006-after.mid \
  --output ../spansynth-results/slakh-flowedit

Synthesis

Synthesise the original score in the selected region, using the surrounding recording as audio context:

spansynth-edit synthesize \
  --audio ../spansynth-inputs/early-slakh-track00006-original.mp3 \
  --midi ../spansynth-inputs/early-slakh-track00006-before.mid \
  --output ../spansynth-results/synthesis

Add --check-inputs to validate audio, MIDI, and timing without loading the model.

Options

Common options are listed below. Flags marked off are enabled by adding them to the command. For all options, run spansynth-edit edit --help or spansynth-edit synthesize --help.

Generation

Inputs and timing

All times are in seconds.

Execution and output

Timing and instruments

Provide aligned audio and MIDI, or use the offsets to align their timelines.

With --crop-start 30, a MIDI note at 36.4 s appears at 6.4 s in the crop. If MIDI time 0 corresponds to the start of that crop, use --midi-offset 30. Set --source-midi-offset independently for the original MIDI. Each run processes one crop, with edit boundaries rounded outward to 40 ms.

The instrument vocabulary lists supported MIDI programs and merged instrument groups. Programs in a group share one model category: for example, 0, 1, 3, 6, and 7 map to Acoustic Piano. Program numbers are zero-based, and MIDI channel 10 selects drums (internal program 128).

Results

All audio outputs are 48 kHz mono. Outside the generated region, output.wav matches input.wav exactly. Use --overwrite to replace existing results.

License and credits

Code and model weights are released under Apache-2.0. See LICENSE and NOTICE for terms and credits, including YourMT3+ and HeartCodec (code license, weight license). Demo recordings retain their original rights.

Training data retain their own licenses, including MAESTRO's CC BY-NC-SA 4.0 terms. The repository's Apache-2.0 license does not replace these dataset terms.

Citation

If you use SpanSynth-Edit in your research, please cite our paper:

@misc{chang2026spansynthedit,
  title={Synthesis and editing of multi-instrument audio mixtures using scalar-quantised latents with {MIDI Span} conditioning},
  author={Sungkyun Chang and Keshav Bhandari and Simon Dixon and Emmanouil Benetos},
  year={2026},
  eprint={2609.25546},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2609.25546}
}