Music-LLM 1B (FMA) — the same text-to-music recipe, 23× more parameters

This is the large sibling of Music-LLM 45M (FMA): 1,027,018,752 parameters (d = 2048, 16 blocks, 16 heads) trained on the same Free Music Archive (FMA medium) data, with the same four-codebook EnCodec token format and the same frozen T5 prompt encoder. Everything is implemented in pure PyTorch, including the two-GPU pipeline parallelism it was trained with.

It is built for research and education, to show what changes, and what does not, when only the model size changes. It is still a small model by music-AI standards: do not expect the quality of large production systems. It produces genre-flavoured musical texture, not finished songs.

What the larger model buys (measured)

The two training runs report validation loss on different held-out sets and with different evaluation code, so their logged numbers (5.362 for this model, 5.410 for the 45M) must not be compared. To compare fairly, both checkpoints were scored on the same 120 FMA clips that neither model trained on (8 per genre, first and last 20-second window of each clip, 960,000 predicted tokens each):

So 23× more parameters bought 0.035 nats — a real but small gain. The models also disagree with each other much more than that gap suggests: KL(1B ‖ 45M) = 0.229 nats per token and Jensen-Shannon divergence = 0.054 (its maximum is 0.693), so they are different predictors of similar accuracy, not one model plus noise.

• Coarse codebook gains most. The cross-entropy gap is 0.043 on the first (coarsest) codebook and 0.025 on the last.
• Genres differ a lot. Gain per genre ranges from +0.104 (electronic) and +0.055 (hip-hop) down to +0.013 (rock), +0.012 (jazz) and +0.006 (instrumental). Across the 15 genres, the gain correlates with how much the two models disagree (r = 0.81).
• The 1B is sharper than it is right. On held-out clips its own entropy (5.16) is lower than its actual cross-entropy (5.32): it is somewhat over-confident, a sign of a model that has started to fit its training clips (see training section). The 45M is the opposite (5.38 vs 5.36).

Measured on 120 clips per model pair, a single run each: treat the per-genre numbers as indicative, not precise.

🔊 Samples

One draw per prompt from this exact checkpoint (30 s, no re-rolls or cherry-picking). Press play:

"blues music"

"classical music"

"electronic music"

"hip-hop music"

"jazz music"

"rock music"

"country music"

"easy listening music"

"experimental music"

"folk music"

"instrumental music"

"international music"

"old-time / historic music"

"pop music"

"soul-rnb music"

A lower loss means the model predicts FMA audio tokens better; it does not by itself mean it sounds better, and no such claim is made. Judge by ear.

Architecture

🚀 How to use

Can transformers.AutoModel load this? No — it is a custom architecture, not a transformers model. Everything needed lives in one file: example.py contains the full model definition and the generation loop, and reads the model size from the checkpoint.

pip install torch numpy transformers sentencepiece soundfile huggingface_hub
hf download khashayargh/music-llm-1b-fma --local-dir music-llm-1b-fma
cd music-llm-1b-fma && python example.py "jazz music" --seconds 15

The weights file is 4.3 GB (float32). Generation needs roughly 6 GB of free GPU or unified memory for the model; it runs on CUDA, Apple Silicon (MPS) or CPU (slowly). transformers is used only to load the frozen T5 and EnCodec components, which download automatically on first run.

Prompt exactly as trained: "<genre> music" in lowercase, for one of blues, classical, country, easy listening, electronic, experimental, folk, hip-hop, instrumental, international, jazz, old-time / historic, pop, rock, soul-rnb. Other wording (for example a bare label like "Blues") is outside what it was trained on.

Training

• Data: FMA medium (Defferrard et al., 2017): 30-second clips in 15 top-level genres (the "Spoken" genre is excluded, it is speech, not music). About 2% of each genre (at least 8 clips) is held out for validation, every held-out clip is scored end to end at every evaluation, and everything else trains on random 20-second windows. In total the model saw about ten times the duration of the training audio.
• Start: random weights. The 45M checkpoints cannot warm-start a model of a different shape, so the loss begins near chance, ln 2048 = 7.62 (the first step measured 8.03), and falls steadily.
• Learning rate was searched, briefly. Six 600-step trials on a log grid from 6e-5 to 6e-4 ended at validation loss 6.749, 6.747, 6.756 and 6.799 (the other two were pruned early). The peak learning rate 9.51e-5 won, narrowly: the three best rates were within 0.01 of each other, so the exact value matters little. Dropout was not searched.
• Recipe: AdamW (weight decay 0.1), 1,000-step warmup then cosine decay to 9.5e-6, batch 32 × 20 s, gradient clipping at 1.0, bfloat16 autocast with float32 master weights and optimizer state.
• Hardware: 2× NVIDIA RTX A6000 with hand-written GPipe-style pipeline parallelism (the 16 blocks split across the two GPUs), gradient accumulation across micro-batches. 12,000 steps took 10.3 hours.
• Validation loss fell at every evaluation to 5.362 at step 12,000, which is the checkpoint released here. The gains had almost stopped (the last three evaluations improved by 0.005, 0.002 and 0.002, as the learning rate reached its floor).
• Gradient norm fell from about 1.3 at the start to a flat floor near 0.27 and stayed there; clipping at 1.0 only ever triggered during the first ~550 steps (the last logged value above 1.0 was at step 554).
• Over-fitting has begun. The gap between loss on held-out clips and on a same-size sample of training clips grew from 0.15 (step 4,000) to 0.25 (step 12,000). With about 0.15 billion audio tokens of data for a model of 1 billion parameters, more steps on this data would mostly memorise it.

Limitations

• Data-limited. FMA medium is far too small for 1B parameters (a common rule of thumb asks for ~20 tokens per parameter; this run saw about 0.14 tokens per parameter of distinct data). That is why the model is only 0.035 nats better than the 45M one on the same clips.
• Coarse prompts. It saw 15 two-word captions, so it cannot follow free-form descriptions.
• Noisy labels. FMA genre tags are assigned by uploaders and are loose, so "pop" or "international" cover very different sounds.
• Licence. FMA tracks carry various Creative Commons licences, many of them non-commercial, so this model is for research and educational use only (CC-BY-NC-4.0).
• Mono, 32 kHz, at most 30 s per generation.

Project

Built by Khashayar Ghamati. The write-up of this model — scaling from 45M to 1B parameters, the training run, the KL-divergence comparison and audio: A Billion-Parameter Music Model: What 23× More Parameters Bought

Related models: music-llm-45m (GTZAN) and music-llm-45m-fma (FMA).

Citation

@misc{ghamati2026musicllm1bfma,
  author       = {Ghamati, Khashayar},
  title        = {Music-LLM 1B (FMA): a one-billion-parameter text-to-music transformer trained on the Free Music Archive},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10846},
  url          = {https://huggingface.co/khashayargh/music-llm-1b-fma},
  howpublished = {\url{https://huggingface.co/khashayargh/music-llm-1b-fma}},
  note         = {Model weights and inference script. Write-up: \url{https://ghamati.com/blog/music-llm-1b/}}
}

Built on

• FMA (training data): Defferrard et al., FMA: A Dataset for Music Analysis, ISMIR 2017. arXiv:1612.01840
• MusicGen (design reference): Copet et al., Simple and Controllable Music Generation, NeurIPS 2023. arXiv:2306.05284
• EnCodec (frozen audio codec): Défossez et al., High Fidelity Neural Audio Compression, 2022. arXiv:2210.13438
• T5 (frozen text encoder): Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR 2020. arXiv:1910.10683
• GPipe (pipeline-parallel training scheme): Huang et al., GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism, NeurIPS 2019. arXiv:1811.06965