CIM-LLaVA1.5-7B-SFT
This model is fine-tuned from LLaVA-1.5-7B using SFT + GRPO with the Cross-modal Identity Mapping (CIM) reward, as described in our CVPR 2026 paper.
Overview
CIM is a reinforcement learning framework that improves image captioning by minimizing information loss during modality conversion. It uses two reward signals — Gallery Representation Consistency (GRC) and Query-gallery Image Relevance (QIR) — to encourage LVLMs to generate fine-grained and precise captions without extra annotations.
Usage
from transformers import LlavaForConditionalGeneration, AutoProcessor
from PIL import Image
model = LlavaForConditionalGeneration.from_pretrained("kkk5/CIM-LLaVA1.5-7B-SFT", torch_dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained("kkk5/CIM-LLaVA1.5-7B-SFT")
image = Image.open("your_image.jpg").convert("RGB")
messages = [{"role": "user", "content": [
{"type": "image"},
{"type": "text", "text": "Caption this image as accurately as possible, without speculation. Describe what you see."},
]}]
text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(images=image, text=text, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
print(output_text)
Citation
@inproceedings{jia2026cross,
title = {Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning},
author = {Jia, Haonan and Dong, Shichao and Dong, Xin and Sun, Zenghui and Wang, Jin and Lan, Jinsong and Zhu, Xiaoyong and Zheng, Bo and Zhang, Kaifu},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages = {766--777},
year = {2026}
}