gme-Qwen2-VL-7B-Instruct

gme-Qwen2-VL-7B

We are excited to present GME-Qwen2VL series of unified multimodal embedding models, which are based on the advanced Qwen2-VL multimodal large language models (MLLMs).

The GME models support three types of input: text, image, and image-text pair, all of which can produce universal vector representations and have powerful retrieval performance.

Key Enhancements of GME Models:

• Unified Multimodal Representation: GME models can process both single-modal and combined-modal inputs, resulting in a unified vector representation. This enables versatile retrieval scenarios (Any2Any Search), supporting tasks such as text retrieval, image retrieval from text, and image-to-image searches.
• High Performance: Achieves state-of-the-art (SOTA) results in our universal multimodal retrieval benchmark (UMRB) and demonstrate strong evaluation scores in the Multimodal Textual Evaluation Benchmark (MTEB).
• Dynamic Image Resolution: Benefiting from Qwen2-VL and our training data, GME models support dynamic resolution image input.
• Strong Visual Retrieval Performance: Enhanced by the Qwen2-VL model series, our models excel in visual document retrieval tasks that require a nuanced understanding of document screenshots. This capability is particularly beneficial for complex document understanding scenarios, such as multimodal retrieval-augmented generation (RAG) applications focused on academic papers.

Developed by: Tongyi Lab, Alibaba Group

Paper: GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

Model List

Usage

Transformers

The remote code has some issues with transformers>=4.52.0, please downgrade or use sentence_transformers

from transformers import AutoModel
from transformers.utils.versions import require_version


require_version(
    "transformers<4.52.0",
    "The remote code has some issues with transformers>=4.52.0, please downgrade: pip install transformers==4.51.3"
)


t2i_prompt = 'Find an image that matches the given text.'
texts = [
    "The Tesla Cybertruck is a battery electric pickup truck built by Tesla, Inc. since 2023.",
    "Alibaba office.",
]
images = [
    'https://upload.wikimedia.org/wikipedia/commons/e/e9/Tesla_Cybertruck_damaged_window.jpg',
    'https://upload.wikimedia.org/wikipedia/commons/e/e0/TaobaoCity_Alibaba_Xixi_Park.jpg',
]


gme = AutoModel.from_pretrained(
    "Alibaba-NLP/gme-Qwen2-VL-7B-Instruct",
    torch_dtype="float16", device_map='cuda', trust_remote_code=True
)


# Single-modal embedding
e_text = gme.get_text_embeddings(texts=texts)
e_image = gme.get_image_embeddings(images=images)
print('Single-modal', (e_text @ e_image.T).tolist())
## Single-modal [[0.279296875, 0.0002658367156982422], [0.06427001953125, 0.304443359375]]

# How to set embedding instruction
e_query = gme.get_text_embeddings(texts=texts, instruction=t2i_prompt)
# If is_query=False, we always use the default instruction.
e_corpus = gme.get_image_embeddings(images=images, is_query=False)
print('Single-modal with instruction', (e_query @ e_corpus.T).tolist())
## Single-modal with instruction [[0.32861328125, 0.026336669921875], [0.09466552734375, 0.3134765625]]

# Fused-modal embedding
e_fused = gme.get_fused_embeddings(texts=texts, images=images)
print('Fused-modal', (e_fused @ e_fused.T).tolist())
## Fused-modal [[1.0, 0.0308685302734375], [0.0308685302734375, 1.0]]

sentence_transformers

The encode function accept str or dict with key(s) in {'text', 'image', 'prompt'}.

Do not pass prompt as the argument to encode, pass as the input as a dict with a prompt key.

from sentence_transformers import SentenceTransformer


t2i_prompt = 'Find an image that matches the given text.'
texts = [
    "The Tesla Cybertruck is a battery electric pickup truck built by Tesla, Inc. since 2023.",
    "Alibaba office.",
]
images = [
    'https://upload.wikimedia.org/wikipedia/commons/e/e9/Tesla_Cybertruck_damaged_window.jpg',
    'https://upload.wikimedia.org/wikipedia/commons/e/e0/TaobaoCity_Alibaba_Xixi_Park.jpg',
]


gme_st = SentenceTransformer("Alibaba-NLP/gme-Qwen2-VL-7B-Instruct")

# Single-modal embedding
e_text = gme_st.encode(texts, convert_to_tensor=True)
e_image = gme_st.encode([dict(image=i) for i in images], convert_to_tensor=True)
print('Single-modal', (e_text @ e_image.T).tolist())
## Single-modal [[0.27880859375, 0.0005745887756347656], [0.06500244140625, 0.306640625]]

# How to set embedding instruction
e_query = gme_st.encode([dict(text=t, prompt=t2i_prompt) for t in texts], convert_to_tensor=True)
# If no prompt, we always use the default instruction.
e_corpus = gme_st.encode([dict(image=i) for i in images], convert_to_tensor=True)
print('Single-modal with instruction', (e_query @ e_corpus.T).tolist())
## Single-modal with instruction [[0.328369140625, 0.0269927978515625], [0.09521484375, 0.316162109375]]

# Fused-modal embedding
e_fused = gme_st.encode([dict(text=t, image=i) for t, i in zip(texts, images)], convert_to_tensor=True)
print('Fused-modal', (e_fused @ e_fused.T).tolist())
## Fused-modal [[0.99951171875, 0.0311737060546875], [0.0311737060546875, 1.0009765625]]

Evaluation

We validated the performance on our universal multimodal retrieval benchmark (UMRB, see Release UMRB) among others.

The MTEB Leaderboard English tab shows the text embeddings performence of our model.

More detailed experimental results can be found in the paper.

Community support

Fine-tuning

GME models can be fine-tuned by SWIFT：

pip install ms-swift -U

# MAX_PIXELS settings to reduce memory usage
# check: https://swift.readthedocs.io/en/latest/BestPractices/Embedding.html
nproc_per_node=8
MAX_PIXELS=1003520 \
USE_HF=1 \
NPROC_PER_NODE=$nproc_per_node \
swift sft \
    --model Alibaba-NLP/gme-Qwen2-VL-7B-Instruct \
    --train_type lora \
    --dataset 'HuggingFaceM4/TextCaps:emb' \
    --torch_dtype bfloat16 \
    --num_train_epochs 1 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps $(expr 64 / $nproc_per_node) \
    --eval_steps 100 \
    --save_steps 100 \
    --eval_strategy steps \
    --save_total_limit 5 \
    --logging_steps 5 \
    --output_dir output \
    --lazy_tokenize true \
    --warmup_ratio 0.05 \
    --learning_rate 5e-6 \
    --deepspeed zero3 \
    --dataloader_num_workers 4 \
    --task_type embedding \
    --loss_type infonce \
    --dataloader_drop_last true

Limitations

• Single Image Input: In Qwen2-VL, an image could be converted into a very large number of visual tokens. We limit the number of visual tokens to 1024 to obtain a good training efficiency. Due to the lack of relevant data, our models and evaluations retain one single image.
• English-only Training: Our models are trained on english data only. Although the Qwen2-VL models are multilingual, the multilingual-multimodal embedding performance are not guaranteed.

We will extend to multi-image input, image-text interleaved data as well as multilingual data in the future version.

Redistribution and Use

We encourage and value diverse applications of GME models and continuous enhancements to the models themselves.

• If you distribute or make GME models (or any derivative works) available, or if you create a product or service (including another AI model) that incorporates them, you must prominently display Built with GME on your website, user interface, blog post, About page, or product documentation.

• If you utilize GME models or their outputs to develop, train, fine-tune, or improve an AI model that is distributed or made available, you must prefix the name of any such AI model with GME.

Cloud API Services

In addition to the open-source GME series models, GME series models are also available as commercial API services on Alibaba Cloud.

• MultiModal Embedding Models: The multimodal-embedding-v1 model service is available.

Note that the models behind the commercial APIs are not entirely identical to the open-source models.

Hiring

We have open positions for Research Interns and Full-Time Researchers to join our team at Tongyi Lab. We are seeking passionate individuals with expertise in representation learning, LLM-driven information retrieval, Retrieval-Augmented Generation (RAG), and agent-based systems. Our team is located in the vibrant cities of Beijing and Hangzhou, offering a collaborative and dynamic work environment where you can contribute to cutting-edge advancements in artificial intelligence and machine learning. If you are driven by curiosity and eager to make a meaningful impact through your work, we would love to hear from you. Please submit your resume along with a brief introduction to dingkun.ldk@alibaba-inc.com.

Citation

If you find our paper or models helpful, please consider cite:

@misc{zhang2024gme,
      title={GME: Improving Universal Multimodal Retrieval by Multimodal LLMs}, 
      author={Zhang, Xin and Zhang, Yanzhao and Xie, Wen and Li, Mingxin and Dai, Ziqi and Long, Dingkun and Xie, Pengjun and Zhang, Meishan and Li, Wenjie and Zhang, Min},
      year={2024},
      eprint={2412.16855},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={http://arxiv.org/abs/2412.16855}, 
}