Model Card for op-hrg — Qwen3-VL-4B-Instruct
Part-level visual grounding model for OP-HRG ("Reasoning-Guided Part-Level Visual Grounding via Reinforcement Learning", ECCV 2026). Given an image and a text query naming an object or a part (e.g. "mug's handle"), it outputs bounding boxes and point.
Model Details
Model Description
- Developed by: Kazi Sajeed Mehrab, Hani Alomari, Najibul Haque Sarker, Chia-Wei Tang, Zaber Ibn Abdul Hakim, Anuj Karpatne, Chris Thomas
- Language(s) (NLP): English
- License: Apache-2.0
- Fine-tuned from model: Qwen3-VL-4B-Instruct
Model Sources
- Paper (arXiv): https://arxiv.org/pdf/2607.15374
- Paper (ECCV 2026 proceedings): https://link.springer.com/chapter/10.1007/978-3-032-37383-0_34
Source Repository Code
- Github Repo: https://github.com/sajeedmehrab/op-hrg
Uses
Direct Use
Given an image and a text query naming an object or a part (e.g. "mug's handle"), the model outputs bounding boxes and points.
Downstream Use
To get masks, pass each (box, point) pair from <answer> to SAM2 (facebook/sam2-hiera-large) or SAM3 (facebook/sam3).
Bias, Risks, and Limitations
Please refer to Paper Section 5.
How to Get Started with the Model
Use the code below to get started with the model.
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
model = Qwen3VLForConditionalGeneration.from_pretrained("ksmehrab/op-hrg-qwen3vl4b-instruct", torch_dtype="bfloat16")
processor = AutoProcessor.from_pretrained("ksmehrab/op-hrg-qwen3vl4b-instruct")
Prompt
training/prompts/ophrg_prompt.txt in the op-hrg repository.
Input and coordinate convention
- Resize the input image to 1024×1024.
- Output coordinates are on a 0–1000 grid. To get pixel coordinates in the original image, multiply x by
original_width / 1000and y byoriginal_height / 1000.
Output format
<locate>reasoning about the query</locate>
<target>object|part</target>
<object_hint>[{"bbox_2d": [x1,y1,x2,y2], "point_2d": [x,y]}, ...] or []</object_hint>
<first_answer>[{"bbox_2d": [x1,y1,x2,y2], "point_2d": [x,y]}, ...]</first_answer>
<criticism>self-check of the first answer. ADJUSTMENT: YES|NO</criticism>
<answer>[{"bbox_2d": [x1,y1,x2,y2], "point_2d": [x,y]}, ...]</answer>
Training Details
Please refer to Paper Section 4.1.
Evaluation
Evaluation code: evaluation/ in the op-hrg repository.
Results
Please refer to Paper Section 4.3.
Citation
If you use this model, please cite:
BibTeX:
@inproceedings{mehrab2026reasoning,
title={Reasoning-Guided Part-Level Visual Grounding via Reinforcement Learning},
author={Mehrab, Kazi Sajeed and Alomari, Hani and Sarker, Najibul Haque and Tang, Chia-Wei and Hakim, Zaber Ibn Abdul and Karpatne, Anuj and Thomas, Chris},
booktitle={European Conference on Computer Vision},
pages={590--608},
year={2026},
organization={Springer}
}
Acknowledgements
This work was supported by the Imageomics Institute, which is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under Award #2118240 (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Model Card Authors
Model Card Contact
ksmehrab [at] vt [dot] edu or DM here