Music-LLM 45M — a compact text-to-music transformer

A 44.9M-parameter MusicGen-style autoregressive transformer that generates music audio from short text prompts, implemented in pure PyTorch — the attention layers, training loop, generation sampler, and the two-GPU pipeline parallelism it was trained with are all hand-written, without training frameworks.

This model was built for research and education — to demonstrate, end to end, how a text-to-music LLM is constructed and trained. It is a small model (45M parameters, 8 hours of training audio, 10 genre captions), so please do not expect the high-quality music of large production models: it generates genre-flavoured musical texture rather than finished songs. Its value is the fully readable, fully explainable implementation and training story.

Newer version: music-llm-45m-fma is this same architecture, continued from these weights and trained on the Free Music Archive (about 25× more clips, 15 genres).

🔊 Samples

Generated by this exact checkpoint (30 s each, CFG 3.0, top-k 250). Press play:

"jazz music"

"metal music"

"classical music"

Download: jazz · metal · classical

Architecture

🚀 How to use

Can transformers.AutoModel load this? No — it is a custom architecture, not a transformers model. Instead, everything needed to run it lives in a single file: example.py contains the full model definition and the generation loop, so three commands get you generating:

pip install torch numpy transformers sentencepiece soundfile huggingface_hub
hf download khashayargh/music-llm-45m --local-dir music-llm-45m
cd music-llm-45m && python example.py "jazz music" --seconds 15

(transformers is used only to load the frozen T5 and EnCodec components, which download automatically on first run.)

Trained prompts: "<genre> music" for blues, classical, country, disco, hiphop, jazz, metal, pop, reggae, rock. Runs on CUDA, Apple Silicon (MPS), or CPU; 15 s of audio takes a few minutes on a laptop.

Training

• Data: GTZAN (999 clips, 10 genres, ~8 h), random 20-second windows
• Hardware: 2× NVIDIA RTX A6000 via hand-written GPipe-style pipeline parallelism (stage 1: embedding + blocks 1–4; stage 2: blocks 5–8 + heads), 4 micro-batches, valid-token-weighted gradient accumulation with test-proven exact parity to single-device training
• Recipe: AdamW, peak lr ≈ 5e-4 (Optuna TPE search run through the real pipeline under the production lr schedule), 1000-step warmup + cosine decay, gradient clipping, non-finite-step guard
• This checkpoint: step 4,000 — the validation-loss minimum (val ≈ 5.98), saved automatically before the train/val curves diverged (early stopping against overfitting on the small corpus)
• War story: the first runs produced infinite gradients that were root-caused to silent PCIe peer-to-peer corruption on the host machine; the pipeline detects this at startup and reroutes its stage boundary through CPU memory at ~0.3% overhead. Hardware-level debugging is part of distributed training.

Limitations

• Small model, small data — by design. Education-scale: expect texture and genre flavour, not melody, structure, or production quality
• Trained on 10 captions total ("blues music" … "rock music") — it cannot follow free-form descriptions; prompt steering is coarse
• GTZAN-derived: research / educational use only (hence CC-BY-NC-4.0)
• Mono, 32 kHz

Project

Built by Khashayar Ghamati. The full story — architecture, the two-GPU pipeline, the PCIe bug hunt, real training curves, and audio to judge: Building a Small Transformer That Makes Music

Citation

If you use this model, please cite it:

@misc{ghamati2026musicllm,
  author       = {Ghamati, Khashayar},
  title        = {Music-LLM 45M: a compact text-to-music transformer},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10847},
  url          = {https://huggingface.co/khashayargh/music-llm-45m},
  howpublished = {\url{https://huggingface.co/khashayargh/music-llm-45m}},
  note         = {Model weights and inference script. Write-up: \url{https://ghamati.com/blog/music-llm/}}
}

Built on

This model uses and builds on the following work; please credit it too:

• MusicGen (design reference): Copet et al., Simple and Controllable Music Generation, NeurIPS 2023. arXiv:2306.05284
• EnCodec (frozen audio codec): Défossez et al., High Fidelity Neural Audio Compression, 2022. arXiv:2210.13438
• T5 (frozen text encoder): Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR 2020. arXiv:1910.10683
• GPipe (pipeline-parallel training scheme): Huang et al., GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism, NeurIPS 2019. arXiv:1811.06965
• GTZAN (training data): Tzanetakis and Cook, Musical genre classification of audio signals, IEEE Transactions on Speech and Audio Processing, 10(5), 2002.