Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models
We introduce Loc3R-VLM, a novel framework that equips 2D VLMs with advanced 3D spatial understanding capabilities from video. Inspired by human cognition, it builds an internal cognitive map of the global environment while explicitly modeling an agent's position and orientation. By jointly capturing global layout and egocentric state, the model excels at two core tasks: language-driven localization and viewpoint-aware 3D reasoning.
This repository hosts the released model checkpoint, which uses LLaVA-Video-7B-Qwen2 as the base model.
More details can be found in our GitHub repository.
Citation
@article{qu2026loc3rvlm,
title={Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models},
author={Kevin Qu and Haozhe Qi and Mihai Dusmanu and Mahdi Rad and Rui Wang and Marc Pollefeys},
journal={arXiv preprint arXiv:2603.18002},
year={2026}
}