UniMambaMia-AV-Qwen2-7B
Audio-visual Video-LLM from Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy (Interspeech 2026, arXiv:2509.17901).
| Component | |
|---|---|
| LLM | Qwen2-7B |
| Vision encoder | SigLIP2-so400m, 384 px, 144 tokens per frame after 2x2 pooling (frozen) |
| Audio encoder | Qwen2-Audio encoder (frozen) |
| Audio compressor | UniMambaMia (causal Mamba2 with gated pooling, 2 layers), 25x compression: one audio token per second |
| Input | 32 frames at 1 fps; the audio tokens of each second follow the visual tokens of that frame |
The weights are stored in float16, the precision used for the evaluations in the paper.
Code, training and evaluation scripts: https://github.com/naver-ai/unimambamia-av
Usage
git clone --recursive https://github.com/naver-ai/unimambamia-av && cd unimambamia-av
bash install.sh
bash eval_unimambamia.sh gwkrsrch/UniMambaMia-AV-Qwen2-7B --tasks av_speakerbench_audiovisual,avqa_hard
License
Apache 2.0, following the licenses of Qwen2-7B, SigLIP2 and the Qwen2-Audio encoder that the model is built from.
Citation
@inproceedings{kim2026doesaudiomatter,
title = {Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy},
author = {Geewook Kim and Minjoon Seo},
booktitle = {Proc. Interspeech 2026},
year = {2026},
url = {https://arxiv.org/abs/2509.17901}
}