Qwen-MusicAVQA-7B
Checkpoints for Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA (arXiv:2608.11329).
A frozen Whisper-large-v3-turbo encoder is grafted onto Qwen2-VL-7B-Instruct through learned linear projections. The same frozen encoder serves two roles: it encodes the video's music track and the user's question, which is posed as synthesized speech rather than text. The language model fuses visual frames, music, and question audio entirely through its pretrained self-attention — there is no cross-modal attention module and no learned fusion network.
The only new modality-specific modules are two linear projectors. Whisper and the Qwen2-VL base weights stay frozen throughout; AVQA Stage 2 additionally trains LoRA adapters (r=64) on the LLM's attention layers.
- Paper: https://arxiv.org/abs/2608.11329
- Code: https://github.com/MKDehdashti/Qwen2-vl-audio
- Developed by: Maryam Dehdashti (Inference Matter Labs)
- Model type: multimodal (video + music audio + spoken question -> generated text)
- Finetuned from: Qwen/Qwen2-VL-7B-Instruct
Not an omni-modal model. There is no text-question path in the deployed configuration and no speech output. The point of the work is that a modular graft is a competitive alternative to omni-modal systems on this task.
Results
MUSIC-AVQA, evaluated on the 7,402-pair available-video test subset (about 20% of the benchmark's videos were no longer downloadable), trained on 8,000 pairs.
| Model | Test acc. |
|---|---|
| Qwen2.5-Omni-7B zero-shot (audio-matched) | 56.82% |
| Qwen2.5-Omni-7B fine-tuned (matched inputs/hyperparameters) | 80.91% |
| PANNs-8 (ablation) | 66.87% |
| PANNs-32 (ablation) | 69.90% |
| Whisper-60s-compressed (ablation) | 70.50% |
| Whisper-30s | 95.91% |
| Qwen-MusicAVQA-7B (Whisper-60s-chunked) | 97.31% |
The headline row is the seed-42 representative run; across three full retraining seeds the result is 96.0% +/- 3.9%. Per-modality for the headline model: Audio 97.4%, Audio-Visual 97.3%, Visual 97.3%.
Central finding. Accuracy tracks how much fine-grained local temporal information the audio representation preserves. At a matched 32-token budget, a stride-pooled Whisper frame sequence beats a globally pooled PANNs vector by 26 points — even though PANNs sees at least as much audio and uses a 50x larger projector.
Robustness. On the rephrased MUSIC-AVQA-R benchmark, accuracy is 96.5% (head) and 95.6% (tail) on 3,000-question samples of each split, providing evidence that accuracy is not solely dependent on the original question templates.
ASR. The same architecture reaches 4.85% WER on the full LibriSpeech test-clean (normalized); that checkpoint is what the AVQA track initializes from.
The fine-tuned Omni comparison matches data, inputs, audio duration, and Stage-2 hyperparameters, but the systems differ in backbone and adaptation; read it as a system-level comparison rather than an isolated encoder comparison.
Published MUSIC-AVQA baselines use the full official splits; these numbers use the available-video subset and 8,000 training pairs, so the two are not directly comparable. See the paper.
Efficiency
Both encoders are frozen and music features are cached, so the complete two-stage AVQA run takes about 5 A100-hours on a single A100 80GB. Stage 1 trains only the 4.6M-parameter music projector (0.05% of the model's parameters) and already reaches 96.0% with the LLM frozen.
Repository layout
The repo root is the merged headline model — Stage-1 projector and Stage-2 LoRA already merged, so it loads in one line. Everything else is organized by track.
(root) Qwen-MusicAVQA-7B, merged - 97.31%
asr/
merged_stage2/ ASR Stage-2 merge (4.85% WER); the AVQA starting point
stage1_only/ ASR Stage 1
lora_stage2/ lora_stage3/ ASR LoRA adapters
avqa/
init/ merged ASR model + untrained music projector (see its README)
headline/
stage1/ music projector trained, LLM frozen - 96.0%
stage2_qproj_frozen/ LoRA, question projector frozen - 97.31% (the root model)
stage2_qproj_tuned/ LoRA, question projector tuned - 95.49%
seeds/seed1234|seed2026/{stage1,stage2}/ the 96.0% +/- 3.9% runs
ablations/
panns8/ panns32/ pooled PANNs vector
whisper32/ whisper32_full/ Whisper-30s, Whisper-60s-compressed
whisper_fullres/ whisper_fullres_varlen/ whisper_fullres_ts/
whisper_fullres_notts/ whisper_fullres_notts_matched/ text-question variants
comparison/qwen2.5-omni/ fine-tuned Qwen2.5-Omni baseline
headline/stage1/ is shared: the two Stage-2 variants were both trained from it, differing only
in whether the question projector stayed frozen. Ablation folder names are the internal W&B
experiment_tag strings; the GitHub
README maps them to the
descriptive configuration names used in the paper (e.g. whisper32 = Whisper-30s,
whisper32_full = Whisper-60s-compressed).
All precomputed audio features are in
MayaKD/qwen2-vl-audio-data —
whisper_features_fullres/ (headline model), whisper_features_full/ (60 s-compressed),
whisper_features/ (30 s), whisper_features_fullres_varlen/, panns_features/, clap_features/,
and tts_questions_r/ (MUSIC-AVQA-R). Download these rather than recomputing: they derive from the
MUSIC-AVQA source videos, which are not redistributed.
Video frames and TTS questions are not published — too large to upload. Regenerate them with
src/avqa/video_precompute.py (needs the source videos) and src/avqa/tts_preprocess.py (needs no
video — the question text ships with the code); see the GitHub README's preprocessing section.
Usage
These checkpoints require the custom transformers fork that adds the Qwen2VLAudio model
classes; they will not load with stock transformers.
git clone --recurse-submodules https://github.com/MKDehdashti/Qwen2-vl-audio
cd Qwen2-vl-audio
pip install -e ./transformers && pip install -e ./qwen-vl-utils
pip install -r requirements.txt
The merged model is at the repo root:
from transformers.models.qwen2_vl.modeling_qwen2_vl import Qwen2VLDualAudioForConditionalGeneration
model = Qwen2VLDualAudioForConditionalGeneration.from_pretrained(
"MayaKD/qwen2-vl-audio", torch_dtype="bfloat16")
Music features are precomputed offline and passed as music_features; question audio is encoded
live by the same frozen Whisper encoder. See the repository README for preprocessing and the
training entry points.
To reproduce training rather than run inference, start from asr/merged_stage2/ — that is the
checkpoint every AVQA run initializes from, not the root.
Limitations
- Evaluated only on MUSIC-AVQA (plus MUSIC-AVQA-R for rephrasing robustness), with a closed 42-answer vocabulary, so exact-match accuracy does not measure open-ended generation.
- The official splits share source videos across train and test, so evaluation measures held-out video-question pairs rather than generalization to unseen videos.
- Questions are delivered as synthesized speech; a text-question variant underperforms and is discussed in the paper.
- Music features are precomputed, so inference on a new video needs a separate encoding pass.
- Each ablation row is a single training run; only the headline configuration has three seeds.
- No music-domain audio encoder (e.g. MERT) was evaluated.
License
- Code (GitHub repository): MIT.
- Model weights: derived from Qwen2-VL-7B-Instruct (Apache-2.0) and Whisper-large-v3-turbo (MIT); the upstream terms carry over to these checkpoints, and the repository is labelled Apache-2.0 accordingly.
- MUSIC-AVQA data is subject to its own terms; no benchmark videos are redistributed here.
Citation
@article{dehdashti2026qwenmusicavqa,
title = {Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA},
author = {Dehdashti, Maryam},
journal = {arXiv preprint arXiv:2608.11329},
year = {2026},
url = {https://arxiv.org/abs/2608.11329}
}
Contact: dehdashti@inferencematter.ai