RefCaptioner

RefCaptioner is an 8B vision-language model for multi-reference image-grounded video captioning. Given a video and an ordered set of reference images, it writes a fluent English description and places tags such as <Image_1> directly after the visual phrases grounded by those references. Reference images may include distractors; the model is expected to omit tags that cannot be grounded in the video.

The released weights are a merged Hugging Face checkpoint based on Qwen3-VL-8B-Instruct. No PEFT adapter is required at inference time.

Task overview

RefCaptioner grounds relevant reference images to local phrases in a video caption while rejecting distractor references.

Model details

Method overview

Training Framework

Installation

The tested environment for loading RefCaptioner and running inference is provided in requirement.txt. Use Python 3.10 and install it with:

pip install -r requirement.txt

Quick start

python inference.py \
  --model TengfeiLiuCoder/RefCaptioner \
  --video /path/to/video.mp4 \
  --images /path/to/ref_1.jpg /path/to/ref_2.jpg /path/to/ref_3.png

The image order defines the tag mapping: the first image is <Image_1>, the second is <Image_2>, and so on. The script uses the exact prompt structure and deterministic decoding configuration used for the reported test-set inference.

Reproduction configuration

The released benchmark captions were generated with the following settings:

The same values are machine-readable in inference_config.json. The eight GPUs used for the full benchmark only provided sample-level parallelism; eight GPUs are not a model requirement. A single device can run the model when it has enough memory, while device_map="auto" can distribute the model across available devices.

Input and output contract

The prompt labels each ordered reference before presenting the associated image, then presents the video. The model must:

1. describe visible video content in one English paragraph;
2. place every used <Image_n> immediately after a concrete grounded phrase;
3. group tags only when they ground the same visual unit; and
4. omit distractor or uncertain references rather than forcing all tags into the caption.

Example output:

MRVBench Results

The following scores report RefCaptioner on MRVBench. Eff-Bind = Ref-Tag-R × Ref-Bind.

These values are tied to the prompt and preprocessing configuration above; changing frame sampling, pixel limits, prompt wording, or decoding can change the results.

Intended use

The model is intended for research on multi-reference video understanding, grounded caption generation, reference selection, and local phrase-to-image binding. It is not designed to identify people, infer sensitive attributes, or establish the provenance or ownership of visual content.

Limitations

• Results depend on frame sampling and may miss short events between sampled frames.
• The output is English and follows a task-specific caption style.
• Reference tags express visual grounding, not identity verification.

Users should verify generated descriptions before applying them in consequential settings.

License

The model weights are released under the Apache 2.0 license, following the base model. The companion benchmark contains separate media and annotation considerations; consult its dataset card before use or redistribution.

Citation

@misc{liu2026refcaptionermultireferenceimagegroundedvideo,
      title={RefCaptioner: Multi-Reference Image-Grounded Video Captioning},
      author={Tengfei Liu and Yang Shi and Yuran Wang and Xiaohan Zhang and Yuqing Wen and Yuqi Tang and Qixun Wang and Zhuoran Zhang and Xuanyu Zhu and Weihong Lin and Xinlei Yu and Yujie Wei and Xinwei Long and Fengxiang Wang and Xinlong Chen and Yue Ding and Jialu Chen and Haotian Wang and Yuanxing Zhang},
      year={2026},
      eprint={2607.28509},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.28509}
}