Model description
xGen-MM-Vid (BLIP-3-Video) is an efficient compact vision-language model (VLM) with an explicit temporal encoder, specifically designed to understand videos. It is developed by Salesforce AI Research. Incorporation of a learanable temporal encoder modules within the original (image-based) BLIP-3 architecture is its key aspect.
Here, we are sharing the 128 token version trained to take 8-frame video inputs. In principle, it is able to take any number of frames, but it was trained with 8-frame videos.
The 32 token version of the same model could be found at: BLIP-3-Video 32 token model.
For more details, check out our tech report. More detailed explanation could also be found in the blog article.
Results
Tokens vs. accuracy
Examples
How to use
Please check out our inference script as an example to use our model. This codebase is based on the xGen-MM.
Bias, Risks, Limitations, and Ethical Considerations
The main data sources are from the internet, including webpages, video stock sites, and curated datasets released by the research community. The model may be subject to bias from the original data source, as well as bias from LLMs and commercial APIs. We strongly recommend users assess safety and fairness before applying to downstream applications.
License
Our code and weights are released under the CC by-NC 4.0 license.
Code acknowledgment
Our code/model is built on top of xGen-MM.
Ethical Considerations
This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety. For further guidance on use cases, refer to our AUP and AI AUP.
Citation
@misc{blip3video-xgenmmvid,
author = {Michael S. Ryoo and Honglu Zhou and Shrikant Kendre and Can Qin and Le Xue and Manli Shu and Silvio Savarese and Ran Xu and Caiming Xiong and Juan Carlos Niebles},
title = {xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs},
year = {2024},
eprint = {2410.16267},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2410.16267},
}
Troubleshoot
- If you missed any packages, please consider the following
pip install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cu121
pip install open_clip_torch==2.24.0
pip install einops
pip install einops-exts
pip install transformers==4.41.1