UGround (The Initial LLaVA-based Version)

Update: We have trained stronger models based on Qwen2-VL with the same data. We suggest using them instead for better performance and more convenient training, inference and deployment.

UGround is a strong GUI visual grounding model trained with a simple recipe. Check our homepage and paper for more details. This work is a collaboration between OSU NLP Group and Orby AI.

• Homepage: https://osu-nlp-group.github.io/UGround/
• Repository: https://github.com/OSU-NLP-Group/UGround
• Paper: https://arxiv.org/abs/2410.05243
• Demo: https://huggingface.co/spaces/orby-osu/UGround
• Point of Contact: Boyu Gou

Models

• Model-V1:
  • Initial UGround:
  • UGround-V1-2B (Qwen2-VL)
  • UGround-V1-7B (Qwen2-VL)
  • UGround-V1-72B (Qwen2-VL)
  • Training Data

Release Plan

• Model Weights
  • Initial Version (the one used in the paper)
  • Qwen2-VL-Based V1 (2B, 7B, 72B)
• Code
  • Inference Code of UGround (Initial & Qwen2-VL-Based)
  • Offline Experiments (Code, Results, and Useful Resources)
    • ScreenSpot
    • Multimodal-Mind2Web
    • OmniAct
    • Android Control
  • Online Experiments
    • Mind2Web-Live-SeeAct-V
    • AndroidWorld-SeeAct-V
  • Data Synthesis Pipeline (Coming Soon)
• Training-Data (V1)
• Online Demo (HF Spaces)

Main Results

GUI Visual Grounding: ScreenSpot (Standard Setting)

GUI Visual Grounding: ScreenSpot (Agent Setting)

Citation Information

If you find this work useful, please consider citing our papers:

@article{gou2024uground,
        title={Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents},
        author={Boyu Gou and Ruohan Wang and Boyuan Zheng and Yanan Xie and Cheng Chang and Yiheng Shu and Huan Sun and Yu Su},
        journal={arXiv preprint arXiv:2410.05243},
        year={2024},
        url={https://arxiv.org/abs/2410.05243},
      }

@article{zheng2023seeact,
        title={GPT-4V(ision) is a Generalist Web Agent, if Grounded},
        author={Boyuan Zheng and Boyu Gou and Jihyung Kil and Huan Sun and Yu Su},
        journal={arXiv preprint arXiv:2401.01614},
        year={2024},
      }