Music-LLM 45M (FMA) — the same compact text-to-music transformer, trained on far more music

This is the second release of Music-LLM 45M: the same 44.9M-parameter architecture, continued from that model's weights and trained on the Free Music Archive (FMA medium) — about 25 times more clips and 15 genres instead of 10. Everything is implemented in pure PyTorch, including the two-GPU pipeline parallelism it was trained with.

It is built for research and education, to show end to end how a text-to-music LLM is constructed and trained. It is a small model: please do not expect the quality of large production systems. It produces genre-flavoured musical texture, not finished songs.

The two validation losses are not comparable: they are measured on different held-out music. A lower loss means the model predicts FMA audio tokens better; it does not by itself mean it sounds better, and I make no such claim. Judge by ear below.

🔊 Samples

One draw per prompt from this exact checkpoint (10 s, fixed seed, CFG 3.0, top-k 250, no re-rolls or cherry-picking). Press play:

"blues music"

"classical music"

"electronic music"

"hip-hop music"

"jazz music"

"rock music"

"country music"

"easy listening music"

"experimental music"

"folk music"

"instrumental music"

"international music"

"old-time / historic music"

"pop music"

"soul-rnb music"

Output loudness varies a lot from genre to genre, and for the prompts I measured (jazz, metal, classical) is several times quieter than the GTZAN model's. A likely reason is that the training clips differ widely in loudness (MP3s from thousands of artists) where GTZAN is uniformly loud. Turn the volume up for the quiet ones.

Architecture

🚀 How to use

Can transformers.AutoModel load this? No — it is a custom architecture, not a transformers model. Everything needed lives in one file: example.py contains the full model definition and the generation loop.

pip install torch numpy transformers sentencepiece soundfile huggingface_hub
hf download khashayargh/music-llm-45m-fma --local-dir music-llm-45m-fma
cd music-llm-45m-fma && python example.py "jazz music" --seconds 15

(transformers is used only to load the frozen T5 and EnCodec components, which download automatically on first run.)

Prompt exactly as trained: "<genre> music" in lowercase, for one of blues, classical, country, easy listening, electronic, experimental, folk, hip-hop, instrumental, international, jazz, old-time / historic, pop, rock, soul-rnb. Other wording (for example a bare label like "Blues") is outside what it was trained on. Runs on CUDA, Apple Silicon (MPS) or CPU.

Training

• Data: FMA medium (Defferrard et al., 2017): 24,882 30-second clips in 15 top-level genres (the "Spoken" genre is excluded, it is speech, not music). After dropping unreadable and too-short files, 24,742 clips are used for training and 120 are held out for validation (8 per genre); training draws random 20-second windows.
• Start: weights of music-llm-45m; the optimizer, learning-rate schedule and step counter start fresh.
• Recipe: AdamW, peak lr 5.23e-4 (found by an Optuna search on the GTZAN model and not re-tuned for FMA), 1,000-step warmup then cosine decay to 5.2e-5, batch 16 × 20 s, gradient clipping at 1.0, float32.
• Hardware: 2× NVIDIA RTX A6000 with hand-written GPipe-style pipeline parallelism (blocks 1–4 on one GPU, 5–8 plus heads on the other). The 20,000 steps took about 25 hours. The model is so small that each GPU peaked at only ~2.5 GB, so the hardware was mostly idle on memory.
• This checkpoint: step 20,000, which was also the validation minimum (val 5.410). The loss was still creeping down by about 0.001 per 200 steps at the end, so more training would help only a little.

Limitations

• Small model, by design. Education-scale: expect texture and genre flavour, not melody, structure or production quality.
• Coarse prompts. It saw 15 two-word captions, so it cannot follow free-form descriptions.
• Noisy labels. FMA genre tags are assigned by uploaders and are loose, so "pop" or "international" cover very different sounds.
• Licence. FMA tracks carry various Creative Commons licences, many of them non-commercial, so this model is for research and educational use only (CC-BY-NC-4.0).
• Mono, 32 kHz.

Project

Built by Khashayar Ghamati. The story of the first model — architecture, the two-GPU pipeline, a PCIe bug hunt, training curves and audio: Building a Small Transformer That Makes Music

Citation

@misc{ghamati2026musicllmfma,
  author       = {Ghamati, Khashayar},
  title        = {Music-LLM 45M (FMA): a compact text-to-music transformer trained on the Free Music Archive},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10848},
  url          = {https://huggingface.co/khashayargh/music-llm-45m-fma},
  howpublished = {\url{https://huggingface.co/khashayargh/music-llm-45m-fma}},
  note         = {Model weights and inference script. Continued from music-llm-45m}
}

Built on

• FMA (training data): Defferrard et al., FMA: A Dataset for Music Analysis, ISMIR 2017. arXiv:1612.01840
• MusicGen (design reference): Copet et al., Simple and Controllable Music Generation, NeurIPS 2023. arXiv:2306.05284
• EnCodec (frozen audio codec): Défossez et al., High Fidelity Neural Audio Compression, 2022. arXiv:2210.13438
• T5 (frozen text encoder): Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR 2020. arXiv:1910.10683
• GPipe (pipeline-parallel training scheme): Huang et al., GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism, NeurIPS 2019. arXiv:1811.06965
• GTZAN (data of the base model): Tzanetakis and Cook, Musical genre classification of audio signals, IEEE Transactions on Speech and Audio Processing, 10(5), 2002.