UI-UX

UI-UX is a multimodal LLM optimized for UI/UX defect diagnosis on mobile interfaces. Built on Qwen3.5 with task-aware GRPO reinforcement learning, it achieves 79.63% on UXBench — surpassing Claude-4.5-Sonnet (65.50%) and GPT-4o class models.

Highlights

• State-of-the-art UX reasoning — 79.63% on UXBench
• UXBench — first vision-language benchmark for UX defect diagnosis, 2,000 samples across 8 tasks
• 8 diagnostic tasks covering Usability, Efficiency, and Trustworthiness
• Thinking mode — chain-of-thought reasoning via <think>...</think> tokens, with reward-based mitigation of overthinking

Model Details

Training data: Multi-task UX datasets — BubbleBtn, BubbleNum, MiniPgm, ModelClose, MultiModel, TextCon, TextOcc, MarkerCons

Quick Start

Installation

pip install transformers accelerate

Inference with Transformers

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "afx-team/UI-UX", dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("afx-team/UI-UX")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "screenshot.png"},
            {"type": "text", "text": """Core Task: Evaluate whether the 'pop-up' offers users an explicit control for closing it.
Options:
A. No modal pop-up is present
B. The modal pop-up lacks an explicit close control
C. The modal pop-up has an explicit close control
Output Format: $\\boxed{X}$ (where X is one of A-C)."""},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

output = model.generate(**inputs, max_new_tokens=8192)
response = processor.decode(output[0][len(inputs.input_ids[0]):], skip_special_tokens=True)
print(response)

Inference with vLLM (Recommended)

vllm serve afx-team/UI-UX --port 8000 --max-model-len 16384 \
    --dtype bfloat16 --enable-reasoning --reasoning-parser deepseek_r1

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
response = client.chat.completions.create(
    model="afx-team/UI-UX",
    messages=[{"role": "user", "content": [...]}],
    max_tokens=8192
)
print(response.choices[0].message.content)

UXBench

UXBench is the first vision-language benchmark for UX defect diagnosis, containing 2,000 VQA samples across 8 tasks in 3 dimensions:

Citation

@inproceedings{uiux2026,
  title={Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach},
  author={Mao, Ruichao and Fang, Zhou and Guo, Teng and Yang, Hao and Li, Yaping and Peng, Shaohua and Huang, Maji and Lin, Xiaoyu and Liu, Shuoyang and Li, Xuepeng and Zhang, Yuyu and Rao, Hai},
  booktitle={CVPR Findings},
  year={2026}
}

License

This project is licensed under the MIT License.

Acknowledgements

UI-UX is built upon Qwen3.5 and trained with ms-swift.