RoboFine-VLM

RoboFine-VLM is a fine-tuned Vision-Language Model for robot manipulation video understanding, built on Qwen3.5-VL-397B-A17B. It was introduced in the paper FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies.

Authors: Xintong Hu, Xuhong Huang, Jinyu Zhang, Yutong Yao, Yuchong Sun, Qiuyue Wang, Mingsheng Li, Sicheng Xie, Yitao Liu, Junhao Chen, Yixuan Chen, Yingming Zheng, Shuai Bai, and Tao Yu.

Model Details

Capabilities

• Robotic Video VQA: Visual question answering on robot manipulation videos (multi-view supported)
• Fine-Grained Manipulation Annotation: Step-by-step captioning across 10 dimensions:

Benchmark Results

Evaluated on RoboFine-Bench, a comprehensive benchmark for robotic video understanding.

VQA (VLM4Robotics Benchmark)

Evaluated on 500 robot manipulation samples across 10 datasets.

Caption (VLM4Robotics Benchmark)

Fine-grained step decomposition of robot manipulation videos.

Settings: fps=4, max_frames=512, temperature=0.0, top_p=0.95, thinking=able(only RoboFine-VLM don't use thinking)

Quick Start

[!Important] RoboFine-VLM is built on Qwen3.5-VL-397B-A17B. Inference efficiency and throughput vary significantly across frameworks. We recommend using the latest framework versions to ensure optimal performance and compatibility.

Using RoboFine-VLM via the Chat Completions API (OpenAI SDK)

After serving the model with a framework like vLLM or SGLang, you can use the following code:

from openai import OpenAI
import httpx

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    http_client=httpx.Client(timeout=httpx.Timeout(600.0, connect=60.0)),
)

# Single-view video (frame URLs as video type)
messages = [{"role": "user", "content": [
    {
        "type": "video",
        "video": [
            "https://your-bucket.oss.aliyuncs.com/frame_0000.jpg",
            "https://your-bucket.oss.aliyuncs.com/frame_0001.jpg",
            # ... more frame URLs
        ],
        "max_frames": 512,
    },
    {"type": "text", "text": "Describe the robot manipulation in this video."},
]}]

response = client.chat.completions.create(
    model="RoboFine-VLM-opensource",
    messages=messages,
    temperature=0.0,
    top_p=0.95,
    max_tokens=32768,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

Hardware Requirements

• 8× H200-141GB: BF16 full precision, no quantization needed
• 8× A100/H100-80GB: Requires FP8 quantization (--quantization fp8)

Training Data

Fine-tuned on FineVLA-Data: 47,159 human-verified trajectories with fine-grained instructions, generated by FineVLA-Tool and validated through human inspection.

Citation

@article{hu2026finevla,
  title={FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies},
  author={Hu, Xintong and Huang, Xuhong and Zhang, Jinyu and Yao, Yutong and Sun, Yuchong and Wang, Qiuyue and Li, Mingsheng and Xie, Sicheng and Liu, Yitao and Chen, Junhao and others},
  journal={arXiv preprint arXiv:2605.27284},
  year={2026}
}