Description
VIBE generates background music for a video that follows an explicit text instruction, not just "music that fits this video", but music that fits and does what the user asked. It extends a tokenizer-free, continuous-latent music model with reinforcement learning against a multimodal LLM judge and rule-based verifiable rewards, so the generated music aligns with both the video and the instruction.
Rather than converting audio to discrete tokens, VIBE models music in a continuous latent space: a multimodal semantic LM produces planning latents, a RITE (Residual Integration Transformer Encoder) stack refines them, and a local diffusion transformer, the LocDiT, denoises under conditional flow matching before a 48 kHz stereo VAE decodes to audio. Video conditioning enters through a CLIP-based encoder, and semantic routing merges the video and instruction signals into a single conditioning stream.
Built on VoxCPM (Apache-2.0), adapted from speech to music.
- Instruction-Aligned Music Generation: conditions on a video and a free-form text instruction, so tempo, key, mood, and instrumentation can be requested explicitly rather than inferred from the video alone.
- Reinforcement Learning from a Multimodal Judge: GRPO against Qwen2.5-Omni-7B, which watches the video and listens to the generated music, scoring musicality, text-music alignment, and video-music alignment. Scores come from constrained digit-logit decoding, so there are no JSON parse failures.
- Hard Verifiable Rewards: rule-based tempo/BPM and musical-key agreement, measured on the generated audio with librosa/essentia and scored against targets parsed from the instruction. Deterministic, CPU-only, no learned parameters, and immune to reward hacking.
- Continuous Latents, No Audio Tokenizer: 48 kHz stereo output through a music VAE, avoiding the quality ceiling and codebook artifacts of discrete tokenization.
Model Architecture
| Semantic LM | MiniCPM4-0.5B architecture: 24 layers, hidden 1024, 16 heads, vocab 73 448 |
| RITE | 8 layers, hidden 1024 |
| LocDiT | 8 layers, conditional flow matching, Euler solver |
| Semantic routing | 8 LocDiT blocks × learned softmax over 24 LM layers |
| Latents | patch size 4, feature dim 64, FSQ latent dim 256 |
| Video encoder | CLIP ViT-B/32, 8 sampled frames, projection dim 512 (frozen, weights embedded) |
| Audio decoder | SongBloom / Stable Audio VAE (frozen, external) |
| Output | 48 kHz stereo |
| Parameters | 953.5 M across 873 tensors (3.51 GB) |
| Precision | bf16 at inference; generator stored fp32, CLIP encoder bf16 |
VIBE is a custom architecture. It is not loadable with transformers.AutoModel;
use the inference scripts in the VIBE repository.
Training
Training data: JamendoMaxCaps (text-to-music pretraining), MusicBench (text-to-music SFT), CMI-Pref (captions for text-to-music preference optimisation), and HarmonySet (video-to-music). Evaluation is on ReelBench.
Provenance
This is the Stage-5 RL policy. The RL LoRA adapter (r=8, α=16, on q_proj and
v_proj of both the LM and the LocDiT) has already been folded into the base
weights using W += (α/r) · B·A, so it loads as a plain model with no LoRA
machinery.
⚠️ Do not pass
--lora_weights_pathagainst this checkpoint. The adapter is already merged; supplying it again would apply it a second time.
Usage
This repository holds weights only, no code. Clone the code repository, then
point --ckpt_dir at a download of this model.
git clone https://github.com/aryanvibhosale/vibe.git && cd vibe
conda env create -f environment.yml && conda activate vibe
Requires
transformers>=5.0.0. The video encoder relies onCLIPModel.get_image_features()returning a pooled-output object, which is v5 behaviour. On transformers 4.x, video-to-music fails withAttributeError: 'Tensor' object has no attribute 'pooler_output'.If environment creation fails on the
transformerspin, install it separately:pip install "transformers>=5.0.0".
Download the weights and the external music VAE:
hf download aryanvibhosale/vibe --local-dir ./vibe_ckpt
hf download rsxdalv/SongBloom \
--include "autoencoder_music_dsp1920.ckpt" "stable_audio_1920_vae.json" \
--local-dir ./music_vae_cache
Video-to-music. Video plus instruction in, music out:
python scripts/infer_v2m.py \
--ckpt_dir ./vibe_ckpt \
--audiovae_path ./music_vae_cache \
--text "An ambient electronic track at 125 BPM in B Minor." \
--video_path /path/to/input_video.mp4 \
--output v2m_out.wav
Text-to-music. Instruction only, no video. Use scripts/infer_ttm.py with the
same --ckpt_dir and --audiovae_path.
Tune quality with --cfg_value (default 2.0) and --inference_timesteps
(default 10 for video-to-music, 20 for text-to-music).
Only the first 10 seconds of
--video_pathare used; 8 frames are sampled uniformly from that window. Longer clips are truncated.
Files in this repository
| File | |
|---|---|
model.safetensors | 3.51 GB, 873 tensors, 953.5 M parameters |
config.json | model configuration |
tokenizer.json, tokenizer_config.json, special_tokens_map.json | LlamaTokenizerFast |
static/ | model card images, not used by the model |
Required external components
| Component | Where | Passed as |
|---|---|---|
| SongBloom / Stable Audio music VAE | rsxdalv/SongBloom (Apache-2.0) | --audiovae_path |
openai/clip-vit-base-patch32 preprocessor config | Hugging Face, fetched at init (a few kB) | n/a |
--audiovae_path is a directory holding autoencoder_music_dsp1920.ckpt and
stable_audio_1920_vae.json. It is deliberately not bundled here: the audio VAE is
never read from the checkpoint directory, and its weights carry their own licence.
The CLIP weights are embedded in model.safetensors. Only the small image
preprocessor config is fetched from the Hub at init, so a populated HF_HOME cache
is enough for offline use. You do not need to download MiniCPM4-0.5B: the
semantic LM weights are in model.safetensors and the tokenizer ships here.
Limitations
- Instruction adherence is strongest for tempo. Automatic musical-key detection is unreliable across the board, including on real reference audio, so key agreement should not be read as a headline metric.
- RL trades distributional metrics for adherence and diversity. The reinforcement-learned policy is not tuned to minimise FAD or maximise ImageBind similarity, and should not be ranked by them or framed as "closest to ground truth".
- Generated audio may be unexpected or contain artifacts, particularly for instructions far from the training distribution. Released for research purposes; we do not recommend production use without further testing.
- Short-form instrumental music. Trained for background music clips, not full
arrangements or vocals. Output is instrumental; the
enlanguage tag refers to the text instruction, not to the audio. - Respect the rights in your inputs. Generating music conditioned on video you do not have rights to, or producing music that imitates a specific artist's protected work, may infringe. We recommend clearly marking generated audio as AI-generated.
License
Released under the Apache-2.0 license.
These weights derive from MiniCPM4-0.5B (Apache-2.0) and embed a frozen CLIP ViT-B/32 (MIT) video encoder. The Qwen2.5-Omni judge and CMI-RM (CMI-RewardBench) shaped training as reward signals only and are not distilled into these weights; neither is redistributed here. CMI-RM's published weights are released for non-commercial use; consult the upstream repository for its current terms. Full third-party attribution is in the NOTICE file. The audio VAE is not included; see Required external components.
Acknowledgements
This work builds on open-source research and code. With thanks to: VoxCPM · MiniCPM-4 · CosyVoice · DAC · Qwen2.5-Omni · MuQ / MuQ-MuLan · CMI-RewardBench