Re3Cap-CIM-Qwen2-VL-7B
This model is fine-tuned from Qwen2-VL-7B-Instruct using GRPO with the Cross-modal Identity Mapping (CIM) reward and the Re³Cap retrieval-guided refinement strategy, as described in our EMNLP 2026 Main paper.
Overview
Re³Cap (Retrieval-Guided Refinement for Image Captioning) is a retrieval-guided reasoning strategy that improves image captioning without requiring additional annotations. It is instantiated by Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA), which together identify hallucinations and omissions in generated captions, leading to more accurate and detailed descriptions.
This checkpoint applies Re³Cap on top of the CIM reward, which rewards the similarity between the images retrieved by the caption and the original image (Jia et al., 2026).
Usage
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
model = Qwen2VLForConditionalGeneration.from_pretrained("kkk5/Re3Cap-CIM-Qwen2-VL-7B", torch_dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained("kkk5/Re3Cap-CIM-Qwen2-VL-7B")
messages = [{"role": "user", "content": [
{"type": "image", "image": "your_image.jpg"},
{"type": "text", "text": "Caption this image as accurately as possible, without speculation. Describe what you see."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
print(output_text)
Citation
@inproceedings{jia2026re3cap,
title = {Re$^3$Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning},
author = {Jia, Haonan and Dong, Shichao and Sun, Zenghui and Zheng, Jiawen and Miao, Ziqi and Shi, Gege and Zhao, Qiuyu and Lan, Jinsong and Zhu, Xiaoyong and Zheng, Bo},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
year = {2026}
}