MIDI-LLM
Built on Llama 3.2 (1B) with an extended vocabulary for MIDI tokens.
Research Paper
- Shih-Lun Wu, Dave Carlton, Ryan Mikayawa, Yoon Kim, Chris Donahue*, and Cheng-Zhi Anna Huang*.
"MIDI-LLM: Improving text-to-MIDI music generation via adapting large language models."
International Society for Music Information Retrieval Conference (ISMIR), 2026.
MIT, Hooktheory, CMU
[Code] [Live Demo] [Paper] [Video]
Model Description
- Base Model:
meta-llama/Llama-3.2-1B - Model Size: 1.4B parameters
- Extended Vocabulary: 183,286 tokens (128,256 for text + 55,030 for MIDI music)
- Architecture:
LlamaForCausalLMwith extended embedding layer - Precision: BFloat16
Quick Start
Clone our Github code repo, run through setup steps, and try:
git clone https://github.com/slSeanWU/MIDI-LLM
cd MIDI-LLM
python generate_transformers.py \
--model slseanwu/MIDI-LLM_Llama-3.2-1B \
--prompt "A cheerful rock song with bright electric guitars" \
--n_outputs 4
The repo and inference scripts provide a more complete usage guide.
Model Details
Extended Vocabulary
The model extends Llama 3.2's vocabulary (128,256 tokens) with 55,030 MIDI tokens representing:
- Onset time (when notes occur)
- Durations (how long each note is held)
- Instrument-pitch pair (which note to play & by which instrument)
These tokens follow the vocabulary of Anticipatory Music Transformer (AMT) (Thickstun et al., TMLR 2024).
Training Data
- Datasets:
- Training objective: Causal language modeling
- Training sequence length: 2,048
- System prompt:
You are a world-class composer. Please compose some music according to the following description: [your input text]
Inference Hyperparameters
Recommended settings for best results:
temperature: 1.0
top_p: 0.98
max_tokens: 2046
Evaluation
This model checkpoint was evaluated with FAD and CLAP metrics on 896 LakhMIDI examples whose IDs can be found in our repo
- https://github.com/slSeanWU/MIDI-LLM/blob/main/assets/evaluation_set_lakh_ids.txt | Model | Params | Precision | FAD ↓ | CLAP ↑ | |-------|--------|-----------|-------|--------| | MIDI-LLM | 1.47B | BF16 | 0.173 | 22.1 | | MIDI-LLM | 1.47B | FP8 | 0.216 | 21.8 |
Citation
If you find our model useful, please cite our research as
@inproceedings{wu2026midillm,
title={{MIDI-LLM}: Improving text-to-{MIDI} music generation via adapting large language models},
author={Wu, Shih-Lun and Carlton, Dave and Mikayawa, Ryan and Kim, Yoon and Donahue, Chris and Huang, Cheng-Zhi Anna},
booktitle={Proc. Int. Soc. for Music Information Retrieval Conference (ISMIR)},
year={2026}
}
License
This model is based on Llama 3.2 and is subject to the Llama 3.2 Community License.