Qwen2-VL-2B-mlx
MLX-format conversion of Qwen/Qwen2-VL-2B, for use with mlx-vlm on Apple Silicon.
Fix included in this repo
The original checkpoint's chat_template.json/chat_template.jinja ships an incomplete Jinja template — it's missing the outer <|im_start|>{role}...<|im_end|> message-wrapping logic, containing only the inner fragment that converts a single message's content list into vision-placeholder tokens. As shipped, calling apply_chat_template() on any real conversation (text-only or with images) returns an empty string, which then causes a crash deep in inference (ValueError: [max] Cannot max reduce over axis 1 with size 0 in get_rope_index).
This repo's chat_template.json/chat_template.jinja have been replaced with the complete, correct Qwen2-VL ChatML template (verified against working sibling checkpoints), so apply_chat_template() works correctly out of the box.
Usage
pip install -U mlx-vlm
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
model_path = "mlx-community/Qwen2-VL-2B-mlx"
model, processor = load(model_path)
config = load_config(model_path)
prompt = apply_chat_template(processor, config, "Describe this image.", num_images=1)
output = generate(model, processor, prompt, image=["path/to/image.jpg"], verbose=False)
print(output)
Original model card
We're excited to unveil Qwen2-VL, the latest iteration of our Qwen-VL model, representing nearly a year of innovation.
[!Important] This is the base pretrained model of Qwen2-VL-2B without instruction tuning.
What's New in Qwen2-VL?
Key Enhancements:
- SoTA understanding of images of various resolution & ratio: Qwen2-VL achieves state-of-the-art performance on visual understanding benchmarks, including MathVista, DocVQA, RealWorldQA, MTVQA, etc.
- Understanding videos of 20min+: Qwen2-VL can understand videos over 20 minutes for high-quality video-based question answering, dialog, content creation, etc.
- Agent that can operate your mobiles, robots, etc.: with the abilities of complex reasoning and decision making, Qwen2-VL can be integrated with devices like mobile phones, robots, etc., for automatic operation based on visual environment and text instructions.
- Multilingual Support: to serve global users, besides English and Chinese, Qwen2-VL now supports the understanding of texts in different languages inside images, including most European languages, Japanese, Korean, Arabic, Vietnamese, etc.
For more information, visit the original Blog and GitHub.
Citation
@article{Qwen2-VL,
title={Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
author={Peng Wang and Shuai Bai and Sinan Tan and Shijie Wang and Zhihao Fan and Jinze Bai and Keqin Chen and Xuejing Liu and Jialin Wang and Wenbin Ge and Yang Fan and Kai Dang and Mengfei Du and Xuancheng Ren and Rui Men and Dayiheng Liu and Chang Zhou and Jingren Zhou and Junyang Lin},
journal={arXiv preprint arXiv:2409.12191},
year={2024}
}