Clef-0.2B
A ~0.2B-parameter decoder-only Transformer for symbolic music generation (MusicXML / MIDI). Given a short conditioning header (genre / ensemble / tempo / key / meter / instrument plan) it generates a multi-track score, and it also supports CONTINUE (extend a given prefix) and INFILL (fill masked bars).
The model was pretrained from scratch on symbolic scores and then instruction-tuned on those three tasks. This is the instruction-tuned checkpoint.
Model summary
| Architecture | decoder-only Transformer, RoPE, RMSNorm, SwiGLU, GQA |
| Params | 201,398,272 (~0.2B) |
| Layers / d_model | 17 / 1024 |
| Attention | 16 query heads : 4 KV heads (GQA), FFN 2816 |
| Context length | 4096 tokens max (trained on 2048-token windows) |
| Tokenizer | tokenizer_v2 — bar-major REMI+, vocab 9507, 24 TPQN |
| Precision | trained in fp8 (torchao) + torch.compile; weights ship bf16 |
Weights format
Inference weights are provided as model.safetensors in Llama
(LlamaForCausalLM) layout with a matching config.json, so they load with
transformers / vLLM directly. This layout was numerically verified against the
native training checkpoint (greedy decode matches token-for-token). Note the
tokenizer is custom (bar-major REMI+, in tokenizer/) — it is not an
AutoTokenizer, so encoding/decoding to MusicXML/MIDI uses this project's
pipeline, not AutoTokenizer.from_pretrained.
Tokenizer / codec
The tokenizer is bar-major REMI+: tracks are interleaved bar by bar so all instruments advance together, which gives the model an explicit notion of simultaneity and helps parts stop together instead of one voice trailing off.
A generation begins with a native conditioning header:
BOS [GENRE] [ENSEMBLE] TEMPO KEY METER <PLAN> {TRACK|DRUM}* <PLAN_END> (BAR ...)* EOS
Drums are encoded by their true General MIDI percussion number (resolved from
<part-list>/midi-instrument/midi-unpitched), and note velocity / dynamics are
preserved. Because drums are stored as real GM voices, decoded output renders
correct kick / snare / hi-hat directly with no display→GM remap.
Prompt example
A generation is conditioned by a header of special tokens; the model then produces
the bars after <PLAN_END>. These are the exact prompts used for the sample files
below.
Rock band — distortion + overdrive guitar, electric bass, drums; A minor, 130 BPM:
<BOS> <GENRE_popular> <ENSEMBLE_band> <TEMPO_130> <KEY_0:minor> <METER_4/4>
<PLAN> <TRACK_30> <TRACK_29> <TRACK_33> <DRUM_TRACK> <PLAN_END>
Piano concerto — piano + string ensemble + cello; E♭ major, 96 BPM:
<BOS> <GENRE_classical> <ENSEMBLE_orchestral> <TEMPO_96> <KEY_-3:major> <METER_4/4>
<PLAN> <TRACK_0> <TRACK_48> <TRACK_42> <PLAN_END>
<TRACK_n>picks a General MIDI program (0 = piano, 30 = distortion guitar, 33 = electric bass, 48 = string ensemble, 42 = cello, …).<DRUM_TRACK>adds a percussion lane.<KEY_k:mode>usesk= signed sharps(+) / flats(−);<TEMPO_n>= BPM;<METER_a/b>= time signature.
Conditioning vocabulary
- GENRE
<GENRE_x>(9):classical·popular·jazz·folk·electronic·soundtrack·world·religious·unknown - ENSEMBLE
<ENSEMBLE_x>(7):solo·duo·band·chamber·orchestral·choir·other - TRACK
<TRACK_n>: any General MIDI program,n= 0–127; plus<DRUM_TRACK>for a percussion lane. - KEY
<KEY_k:mode>:k= −7…+7 (sharps/flats),mode∈major·minor·dorian·phrygian·lydian·mixolydian·none. - TEMPO
<TEMPO_n>: integer BPM (5–999). - METER
<METER_a/b>: time signature (e.g.4/4,3/4,6/8).
(The genre/ensemble labels are coarse; unknown / other are valid and common.)
Samples
samples/promo_rock.mid and samples/promo_piano_concerto.mid are ~3-minute
pieces generated from the two prompts above (best-of-N seeds, CONTINUE-stitched
past the context window). Render with any General MIDI SoundFont.
Training
- Pretraining: from scratch, 3 epochs (13,821 steps), global batch 256, fp8 +
torch.compile. Data: ~1.18M bar-major windows built from the public sources listed below, with ×12 transpose augmentation. - Instruction tuning (this checkpoint): GENERATE / CONTINUE / INFILL tasks mixed with in-key curation (in-key ≥ 0.88) and a base-replay set to limit forgetting. 760 steps (2 epochs), best val loss 0.3088 @ step 299 (adopted). Hardware: 1× RTX PRO 6000 Blackwell (96 GB), ~80k tok/s, ~45 min.
Evaluation
Automatic metrics on held-out generation prompts (2 seeds). Values are absolute for this model; real human-composed scores are given only as a rough anchor.
| metric | Clef-0.2B | real scores (ref) |
|---|---|---|
| in-key ratio ↑ | 0.81 | ~0.96 |
| dissonance ↓ | 0.29 | ~0.15 |
| co-termination spread ↓ | 0.14 | — |
| early part death ↓ | 0.10 | — |
Task success (model emits ≥ 1 new valid bar for the task):
| task | success |
|---|---|
| CONTINUE | 12/12 |
| INFILL | 12/12 |
- in-key ratio — fraction of notes inside the conditioned key.
- dissonance — ratio of dissonant simultaneous intervals.
- co-termination spread / early part death — how tightly instruments end together vs. voices dropping out early (lower is better).
How to generate
Generation runs in this project's pipeline (bar-major decode → MusicXML/MIDI), then any General MIDI SoundFont (e.g. MuseScore) renders audio.
Recommended defaults: temperature 0.7, top_k 24, repetition_penalty ~1.12.
This is the setting used for the sample pieces and is a good starting point.
Optional (supported when decoding with vLLM / transformers): top_p 0.95 and
presence_penalty 0.15 for extra variety. ⚠️ Do not use frequency_penalty — it
destabilizes the structural tokens (BAR / TRK / PLAN). For pieces longer than the
context window, continue past it with a slightly lower continuation temperature
(~0.76) to soften seams. Optionally post-process drums with a sanitize_drums
step that snaps rare out-of-kit hits (e.g. GM 71/72) back to a standard kit.
Limitations
- Context is 4096 tokens (~35 bars). Longer pieces are produced by stitching CONTINUE windows, which can cause a slight mood shift at the seam. A long-context retrain (or RoPE scaling at inference) would fix this properly.
- Dissonance stays somewhat above real-score levels.
- Research artifact; quality is not guaranteed for production use.
Training data
Built from the following public symbolic-music sources:
- PDMX — https://github.com/pnlong/PDMX
- KernScores / Humdrum — https://github.com/humdrum-tools/humdrum-data · https://kern.humdrum.org/
- OpenScore Lieder — https://github.com/OpenScore/Lieder
- The Session — https://github.com/adactio/TheSession-data
License
Released under Apache-2.0.