GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives

English | 简体中文

🔗 Quick Links

• 🚀 Online Demo: Hugging Face Spaces.
• 🌐 Project Page: GroundingPI.
• 💻 GitHub Code: groundingpi/GroundingPI.
• 📄 Paper: arXiv:2609.39601.

Model Overview

Description:

Precise grounding matters. It specifies which object is the target and where that object is, even in clutter and for tiny objects, and it has to be fast enough for closed-loop control. Yet vision-language-action (VLA) and world-action models (WAMs) take perception from general-purpose vision-language and video-generation backbones, which still fail in these settings.

We introduce GroundingPI, a 4B grounding foundation model that generates points and boxes as quantized coordinates in a shared vocabulary. Against 44 baselines across 34 grounding benchmarks spanning 11 perceptual capabilities, GroundingPI establishes a new state of the art, averaging 73.68%, above the larger GPT-6 Astra (71.54%). As a downstream visual backbone, GroundingPI improves performance on robotic manipulation and autonomous driving. On RoboTwin 2.0, it outperforms every mainstream backbone evaluated in all four out-of-distribution settings, by up to 24.8% relative to the strongest backbone. On nuScenes, used as the visual backbone, GroundingPI attains an average open-loop L2 error of 0.296 m.

These results support grounding as a perceptual foundation, and dedicated perceptual pretraining as a promising direction for foundation models of physical intelligence. This repository releases the autoregressive visual-language checkpoint used for grounding; downstream action-policy results are described in the paper.

Demo Videos

License/Terms of Use:

Our original contributions are available under Apache 2.0, with no additional restrictions imposed by this project. Third-party material retains its applicable licenses, including the Kimi K3 License for Kimi-derived material and applicable derivative works. Its conditions continue to apply when using or redistributing the combined model package. See LICENSE for the scope and full license texts.

Deployment Geography:

Global.

Use Case:

• Open-vocabulary object localization and dense-scene grounding.
• Referring-expression comprehension and visual-prompt-based localization.
• Point-based localization, spatial reasoning, and GUI element grounding.
• OCR with text localization and document layout understanding.
• Perception research for robotics, embodied agents, and autonomous systems.

Release Date:

• Webpage [10/04/2026]: Project webpage released: GroundingPI.
• Code [10/03/2026]: GitHub code released: GroundingPI.
• Paper [09/30/2026]: GroundingPI.

@misc{yu2026groundingpigroundingfoundationmodel,
  title = {{GroundingPI}: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives},
  author = {Qize Yu and Lianrui Fan and Boyu Chen and Jiaqi Liang and Xini Ding and Yue Chen and Zetian Song and Yuran Wang and Yi Zou and Kaixuan Wang and Tianxing Chen and Wenxuan Song and Bohan Zhou and Mingleyang Li and Siqiao Huang and Yuqi Ye and Caigao Jiang and Wei Wei and Ruihai Wu and Hang Zhang and Yixiao Ge and Shuchang Zhou and Shilong Liu and Xianming Liu and Ping Luo and Shiyu Huang},
  year = {2026},
  eprint = {2609.39601},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2609.39601},
}

Model Architecture:

Architecture type: autoregressive vision-language model with a shared vocabulary for semantic labels, protocol markers, points, and boxes.

• Vision encoder: MoonViT-V2 / Kimi-K3 vision backbone.
• Language backbone: Qwen3-4B-Instruct-2507.
• Multimodal projector: 2 × 2 spatial aggregation followed by a two-layer MLP.
• Output interface: 1,000 quantized coordinate tokens together with variable-length semantic text.
• Model family: 4B language backbone; custom GroundingPIForConditionalGeneration implementation.

Input(s):

Input types: image and text.

• Image: one RGB image per request. The Python client accepts JPEG, PNG, and WebP files; the supplied processor handles image preparation.
• Text: a natural-language instruction, category list, referring expression, OCR/layout query, or a prompt containing example boxes.
• Image encoding for the HTTP API: an image_url content part containing a base64 data URI, alongside a text content part.

Use the checkpoint's own tokenizer, processor, and chat template. Multiple categories are separated by </c>. Reference boxes use the same 0–999 spatial-token vocabulary as outputs.

Output(s):

Output type: text containing semantic labels and quantized spatial coordinates.

All three released models use the GAM protocol: integer coordinate tokens <0> through <999>, object-reference delimiters, and box delimiters. A bounding box contains (x1, y1, x2, y2); a point contains (x, y). Adjacent coordinate tokens have no intervening spaces. Multiple instances of the same label are comma-separated inside one box wrapper. A missing target is represented by None.

Illustrative syntax:

<|object_ref_start|>car<|object_ref_end|><|box_start|><100><200><500><650><|box_end|>
<|object_ref_start|>car center<|object_ref_end|><|box_start|><300><425><|box_end|>
<|object_ref_start|>absent object<|object_ref_end|><|box_start|>None<|box_end|>

The client returns parsed predictions together with raw_output, finish_reason, usage, parse_error, and valid. It maps coordinates back to image pixels for visualization. A truncated or malformed response is marked invalid. For custom API clients, preserve spatial tokens with skip_special_tokens=false and avoid inserting spaces between them.

Software Integration:

Runtime engines: the supplied vLLM serving integration and a native Transformers reference service.

Default serving environment: Linux, Python 3.12, a compatible platform Torch/vLLM 0.18.x runtime, and the repository's bundled Transformers 5.7.0 fork. The installer prepares the matching environment and model adapter. Use the supplied setup workflow; installing arbitrary upstream package versions is not an equivalent configuration.

Tested hardware: NVIDIA B300, B200, H200, H800, and PPU. Runtime packages must match the accelerator. For PPU, select the platform during setup in the matching vendor runtime image.

The default vLLM service uses BF16, one image per request, a 16,384-token context limit, tensor parallel size 1, and eager execution. These are the supplied serving defaults, not a promise that every hardware/backend combination has identical behavior.

Model Version(s):

Evaluation

The evaluation toolkit supports 7 modes:

All three released checkpoints use GAM mode.

Evaluation code and instructions: GitHub.

Quantitative Evaluation Benchmarks

Visual Grounding

Physical Intelligence

Inference:

Installation

Run these commands from the GroundingPI source repository root after obtaining the code package. Start from Linux with Python 3.12 and a compatible vLLM 0.18.x / Torch accelerator runtime.

python3 -m pip install -r requirements.txt huggingface_hub
hf download GroundingPI/GroundingPI --local-dir weights/vlm

# NVIDIA GPU environment
python3 run.py setup serve
python3 run.py serve

For PPU, replace the GPU setup command with the following in the matching platform image; launch with the same command:

python3 run.py setup serve --platform ppu
python3 run.py serve

Setup creates a separate serving environment that inherits the platform runtime, installs the bundled Transformers fork, and records the platform choice. It expects the platform runtime to be present already. Retain the complete model package when downloading. The launcher's adapter is written to a separate overlay and reuses the original weight shards.

Worker (recommended)

Keep the model service running and reuse the HTTP client:

from grounding_pi import GroundingPi, visualize

client = GroundingPi(
    base_url="http://127.0.0.1:8000/v1",
    model="groundingpi",
)
result = client.predict("example.jpg", "the red car", task="bbox")
print(result.to_dict())
if result.valid:
    visualize("example.jpg", result).save("prediction.png")

point_result = client.predict(
    "example.jpg", "the center of the red car", task="point"
)

The client is lightweight and does not load model weights. The service performs inference and retains the model across requests.

Usage Example

python examples/predict.py \
  --image example.jpg --phrase "the red car" \
  --task bbox --output outputs/car

Supported Tasks & Prompt Templates

Given reference boxes <|box_start|><100><200><500><650><|box_end|> indicating one or more objects, find all similar objects in the image and output their bounding boxes.

The convenience client's predict() method wraps referring-box and referring-point prompts. Use an OpenAI-compatible request to /chat/completions for the other task templates; keep the image and prompt in the same user message.

Generation Modes

GroundingPI uses autoregressive generation. The interactive client uses greedy sampling by default.

Inference Infrastructure

The provided vLLM path manages model execution and KV caching through a custom Transformers adapter. It normalizes model configuration and processor handling while preserving the native spatial output vocabulary. Use a new overlay output directory when switching checkpoint revisions.

The current release launcher explicitly selects eager execution. CUDA Graph is not enabled by this GroundingPI recipe. GroundAnything's SGLang, entropy-guided, and self-speculative paths belong to that model's separate runtime integration.

The native reference service is available to an already prepared reference environment through configs/release/vlm_serve.yaml; the vLLM workflow above is the documented serving entrypoint.

Ethical Considerations:

Grounding predictions can miss small or occluded objects, repeat instances, or produce inaccurate text and coordinates. Validate localization quality on the intended task and inspect incomplete responses. GUI points describe image locations; the model does not execute interface actions. Perception outputs require task-specific validation before integration into physical systems.

GroundingPI：以视觉基元迈向物理智能的视觉定位基础模型

English | 简体中文

🔗 快速链接

• 🚀 在线演示： Hugging Face Spaces。
• 🌐 项目主页： GroundingPI。
• 💻 GitHub 代码： groundingpi/GroundingPI。
• 📄 论文： arXiv:2609.39601。

模型概览

模型介绍

精确的视觉定位至关重要。它需要明确目标是哪个物体、物体位于何处，即使面对杂乱场景和微小目标也能完成定位，同时还要足够快速，以满足闭环控制的需求。然而，视觉—语言—动作模型（VLA）和世界—动作模型（WAM）的感知能力依赖通用视觉语言与视频生成骨干网络，这些骨干网络在上述场景中仍有不足。

我们提出 GroundingPI，一个 4B 视觉定位基础模型，通过共享词表将点和框生成为量化坐标。在覆盖 11 种感知能力的 34 项视觉定位基准上，与 44 个基线进行比较，GroundingPI 取得了新的最佳表现，平均成绩为 73.68%，超过规模更大的 GPT-6 Astra（71.54%）。作为下游任务的视觉骨干网络，GroundingPI 能够提升机器人操作与自动驾驶的表现。在 RoboTwin 2.0 上，它在全部四种分布外设置中均优于所有参评的主流骨干网络，相对最强骨干网络的提升最高达 24.8%。在 nuScenes 上，以 GroundingPI 作为视觉骨干网络时，平均开环 L2 误差为 0.296 m。

这些结果表明，视觉定位可以成为感知的基础，而专门的感知预训练是构建物理智能基础模型的一个有前景的方向。本仓库发布用于视觉定位的自回归视觉语言模型权重；下游动作策略的结果详见论文。

演示视频

许可证与使用条款

本项目的原创贡献采用 Apache 2.0 许可证，项目不附加其他限制。第三方材料保留其适用的许可证，其中源自 Kimi 的材料及适用的衍生作品受 Kimi K3 License 约束。使用或再分发完整模型包时，其相关条件仍然适用。许可范围及完整许可文本见 LICENSE。

部署地区

全球。

适用场景

• 开放词汇目标定位与密集场景视觉定位。
• 指代表达理解与基于视觉提示的定位。
• 点定位、空间推理与 GUI 元素定位。
• 结合文本定位的 OCR 与文档版面理解。
• 面向机器人、具身智能体和自主系统的感知研究。

发布日志

• 项目网页 [10/04/2026]： 项目网页已发布：GroundingPI。
• 代码 [10/03/2026]： GitHub 代码已发布：GroundingPI。
• 论文 [09/30/2026]： GroundingPI。

@misc{yu2026groundingpigroundingfoundationmodel,
  title = {{GroundingPI}: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives},
  author = {Qize Yu and Lianrui Fan and Boyu Chen and Jiaqi Liang and Xini Ding and Yue Chen and Zetian Song and Yuran Wang and Yi Zou and Kaixuan Wang and Tianxing Chen and Wenxuan Song and Bohan Zhou and Mingleyang Li and Siqiao Huang and Yuqi Ye and Caigao Jiang and Wei Wei and Ruihai Wu and Hang Zhang and Yixiao Ge and Shuchang Zhou and Shilong Liu and Xianming Liu and Ping Luo and Shiyu Huang},
  year = {2026},
  eprint = {2609.39601},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2609.39601},
}

模型架构

架构类型： 自回归视觉语言模型，语义标签、协议标记、点和框共用同一词表。

• 视觉编码器： MoonViT-V2 / Kimi-K3 视觉骨干网络。
• 语言骨干网络： Qwen3-4B-Instruct-2507。
• 多模态投影器： 2 × 2 空间聚合，随后接两层 MLP。
• 输出接口： 1,000 个量化坐标 token 与可变长度的语义文本。
• 模型系列： 4B 语言骨干网络；采用自定义的 GroundingPIForConditionalGeneration 实现。

输入

输入类型： 图像与文本。

• 图像： 每次请求输入一张 RGB 图像。Python 客户端接受 JPEG、PNG 和 WebP 文件；配套处理器负责图像预处理。
• 文本： 自然语言指令、类别列表、指代表达、OCR／版面查询，或包含示例框的提示词。
• HTTP API 中的图像编码： 将 base64 data URI 放入 image_url 内容项，与 text 内容项一同传入。

请使用该模型权重自带的 tokenizer、processor 和对话模板。多个类别以 </c> 分隔。参考框与输出共用 0–999 的空间 token 词表。

输出

输出类型： 包含语义标签和量化空间坐标的文本。

本次发布的三个模型均使用 GAM 协议：整数坐标 token <0> 至 <999>，以及对象引用分隔符和框分隔符。边界框包含 (x1, y1, x2, y2)，点包含 (x, y)。相邻坐标 token 之间没有空格。同一标签的多个实例放在同一组框分隔符内，以逗号分隔。未找到的目标以 None 表示。

格式示例：

<|object_ref_start|>car<|object_ref_end|><|box_start|><100><200><500><650><|box_end|>
<|object_ref_start|>car center<|object_ref_end|><|box_start|><300><425><|box_end|>
<|object_ref_start|>absent object<|object_ref_end|><|box_start|>None<|box_end|>

客户端返回解析后的预测结果，以及 raw_output、finish_reason、usage、parse_error 和 valid。可视化时会将坐标映射回图像像素。被截断或格式错误的响应会被标记为无效。自定义 API 客户端时，请使用 skip_special_tokens=false 保留空间 token，并避免在其间插入空格。

软件集成

运行引擎： 配套的 vLLM 服务集成与原生 Transformers 参考服务。

默认服务环境： Linux、Python 3.12、与硬件平台兼容的 Torch / vLLM 0.18.x 运行时，以及仓库内置的 Transformers 5.7.0 分支。安装程序会配置匹配的环境与模型适配器。请使用配套的安装流程；任意安装上游软件包版本并不等同于该配置。

已测试硬件： NVIDIA B300、B200、H200、H800，以及 PPU。运行时软件包必须与加速器匹配。对于 PPU，请在对应厂商的运行时镜像中执行安装，并选择相应平台。

默认 vLLM 服务采用 BF16，每次请求输入一张图像，上下文长度上限为 16,384 个 token，张量并行度为 1，并使用 eager execution（即时执行）。这些是配套服务的默认设置，并不保证所有硬件与后端组合的行为完全一致。

模型版本

评测

评测工具支持 7 种模式：

本次发布的三个模型均使用 GAM 模式。

评测代码与使用说明：GitHub。

定量评测结果

视觉定位

物理智能

推理

安装

获取代码包后，在 GroundingPI 源码仓库根目录执行以下命令。基础环境应为 Linux、Python 3.12，以及兼容的 vLLM 0.18.x / Torch 加速器运行时。

python3 -m pip install -r requirements.txt huggingface_hub
hf download GroundingPI/GroundingPI --local-dir weights/vlm

# NVIDIA GPU environment
python3 run.py setup serve
python3 run.py serve

对于 PPU，请在对应平台镜像中将 GPU 安装命令替换为以下命令；启动命令相同：

python3 run.py setup serve --platform ppu
python3 run.py serve

安装流程会创建独立的服务环境，继承平台运行时，安装内置的 Transformers 分支，并记录所选平台。执行前应已具备平台运行时。下载时请保留完整模型包。启动器会将适配器写入单独的覆盖层目录，并复用原始权重分片。

常驻服务（推荐）

保持模型服务运行，通过 HTTP 客户端重复调用：

from grounding_pi import GroundingPi, visualize

client = GroundingPi(
    base_url="http://127.0.0.1:8000/v1",
    model="groundingpi",
)
result = client.predict("example.jpg", "the red car", task="bbox")
print(result.to_dict())
if result.valid:
    visualize("example.jpg", result).save("prediction.png")

point_result = client.predict(
    "example.jpg", "the center of the red car", task="point"
)

客户端轻量运行，不加载模型权重。推理由服务端执行，模型会在请求之间保持加载状态。

使用示例

python examples/predict.py \
  --image example.jpg --phrase "the red car" \
  --task bbox --output outputs/car

支持的任务与提示词模板

Given reference boxes <|box_start|><100><200><500><650><|box_end|> indicating one or more objects, find all similar objects in the image and output their bounding boxes.

便捷客户端的 predict() 方法封装了指代框定位与指代点定位的提示词。其他任务模板可通过兼容 OpenAI 的请求调用 /chat/completions；请将图像与提示词放在同一条用户消息中。

生成模式

GroundingPI 使用自回归生成。交互式客户端默认采用贪心采样。

推理基础设施

配套的 vLLM 路径通过自定义 Transformers 适配器管理模型执行与 KV 缓存。适配器统一模型配置与处理器的使用方式，同时保留原生空间输出词表。切换模型权重版本时，请使用新的覆盖层输出目录。

当前发布版本的启动器明确使用 eager execution（即时执行）。此 GroundingPI 配置未启用 CUDA Graph。GroundAnything 的 SGLang、熵引导解码与自推测解码路径属于该模型独立的运行时集成。

已配置好的参考环境可通过 configs/release/vlm_serve.yaml 使用原生参考服务；上文的 vLLM 流程是文档所说明的服务入口。

伦理考量

视觉定位预测可能漏检微小或被遮挡的物体、重复输出实例，或生成不准确的文本与坐标。请在目标任务上验证定位质量，并检查不完整的响应。GUI 点输出描述的是图像中的位置；模型本身不执行界面操作。将感知输出集成到物理系统之前，需要针对具体任务进行验证。