UI-Genie-Agent-3B
Model Description
UI-Genie-Agent-3B is a state-of-the-art Multimodal Large Language Model specifically trained for mobile GUI automation tasks. It is part of the UI-Genie framework, which introduces a novel self-improving approach for enhancing MLLM-based mobile GUI agents through iterative agent-reward model co-evolution.
This model achieves state-of-the-art performance on mobile GUI benchmarks by eliminating the need for manual annotation through synthetic trajectory generation guided by our specialized reward model UI-Genie-RM.
Model Architecture
- Base Model: Qwen2.5-VL-3B-Instruct
- Training Method: Supervised fine-tuning with exisiting trajetory datasets and our synthetic trajectory data
- Action Space Coverage: Supports comprehensive mobile interactions (click, swipe, type, etc.) and Set-of-Mark mode.
Performance
AndroidControl Benchmark
| Model Size | Low-Level Tasks | High-Level Tasks |
|---|---|---|
| UI-Genie-Agent-3B | 93.8% SR | 72.9% SR |
| UI-TARS-2B | 89.3% SR | 68.9% SR |
| Qwen2.5-VL-3B | 90.8% SR | 63.7% SR |
AndroidLab Benchmark
| Model | Success Rate | Sub-Goal Success Rate |
|---|---|---|
| UI-Genie-Agent-3B | 28.8% | 35.4% |
| AutoGLM | 36.2% | - |
| Qwen2.5-VL-7B | 14.9% | 18.7% |
Training Data
Our model is trained on a combination of:
- AndroidControl: 15.3K trajectories (high & low level tasks)
- AMEX: 2.9K trajectories (high-level tasks)
- AndroidLab: 726 trajectories (high-level tasks)
- UI-Genie-Agent-16k: 2.2K synthetic trajectories (our generated data)
Action Space
The model supports a comprehensive action space for mobile interactions:
| Action Type | Parameters | Description |
|---|---|---|
open | app_name, action_desc | Launch applications |
click | coordinate/som, action_desc | Tap UI elements |
swipe | coordinate/som, direction, distance, action_desc | Scroll the screen |
long_press | coordinate/som, action_desc | Long press interactions |
type | text, action_desc | Text input |
system_button | button, action_desc | System button presses |
wait | time, action_desc | Wait operations |
terminate | status, action_desc | Task completion |
Citation
@misc{xiao2025uigenieselfimprovingapproachiteratively,
title={UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents},
author={Han Xiao and Guozhi Wang and Yuxiang Chai and Zimu Lu and Weifeng Lin and Hao He and Lue Fan and Liuyang Bian and Rui Hu and Liang Liu and Shuai Ren and Yafei Wen and Xiaoxin Chen and Aojun Zhou and Hongsheng Li},
year={2025},
eprint={2505.21496},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2505.21496},
}