LLaDA-UI is an MoE-based, block-wise diffusion vision-language GUI agent. It understands screenshots at their native aspect ratio and produces grounded coordinates or structured actions for mobile, desktop, and web interfaces.
This repository contains the model weights. The inference code, example requests, and SGLang serving instructions are available in the LLaDA-UI code repository.
Highlights
- Diffusion-native GUI agent: reasons about screenshots and generates GUI actions with block-wise diffusion decoding rather than autoregressive token-by-token decoding.
- Mixture-of-Experts architecture: approximately 16.7B total parameters.
- Native-resolution vision: preserves the aspect ratio and visual details of mobile, desktop, and web screenshots.
- Cross-platform interaction: supports GUI grounding and agent tasks on mobile, desktop, and web interfaces.
- Structured output: emits normalized coordinates or tagged reasoning and executable actions.
Model Details
| Item | Description |
|---|---|
| Model type | MoE block-wise diffusion vision-language GUI agent |
| Total parameters | Approximately 16.7B |
| Language backbone | LLaDA2.0-mini-base |
| Vision encoder | Native-resolution ViT initialized from SigLIP, with 2D RoPE |
| Output | Grounded coordinates, text reasoning, and structured GUI actions |
| Coordinate convention | Integer coordinates normalized to [0, 999] |
| Supported domains | GUI grounding, mobile, desktop, and web |
| Weight format | BF16 Safetensors |
Inference Pipeline
Benchmark Results
Quick Start
1. Clone the inference code
git clone https://github.com/inclusionAI/LLaDA-UI.git
cd LLaDA-UI
2. Create an environment
The standalone Hugging Face inference path has been tested with Python 3.10, PyTorch 2.5.1, Transformers 4.51.0, and FlashAttention 2.7.4.post1.
conda create -n llada-ui python=3.10 -y
conda activate llada-ui
pip install torch==2.5.1 torchvision \
--index-url https://download.pytorch.org/whl/cu124
pip install transformers==4.51.0 Pillow numpy einops accelerate \
sentencepiece protobuf safetensors
pip install ninja
pip install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
FlashAttention must be built against a compatible CUDA and PyTorch setup. The checkpoint is about 32 GB in BF16; additional GPU memory is required for model execution and visual tokens.
3. Run single-image grounding
The following command downloads the checkpoint from Hugging Face on first use and returns the center point of the requested UI element:
CUDA_VISIBLE_DEVICES=0 IMAGE_MAX_PIXELS=12845056 \
python -u inference/inference_hf.py \
--ckpt inclusionAI/LLaDA-UI \
--image /path/to/screenshot.png \
--prompt "click the search button" \
--gen-length 32 \
--steps 32 \
--block-length 32
Example output:
GENERATION: [742,186]
POINT(0~1): [0.742, 0.186]
The raw model coordinate is normalized to [0, 999]. The inference script
also prints the corresponding [0, 1] coordinate. To convert it to screen
pixels for an image of width W and height H, use (x * W, y * H) with the
[0, 1] values.
Large screenshots may also require increasing --max-length beyond its
default value of 8192.
SGLang Serving
For agent evaluation or an OpenAI-compatible endpoint, follow the SGLang server guide. The server uses LLaDA-UI-specific multimodal diffusion patches and cannot be started from a stock SGLang installation alone.
After installing the documented serving environment, launch it with:
huggingface-cli download inclusionAI/LLaDA-UI \
--local-dir /path/to/LLaDA-UI-checkpoint
CUDA_VISIBLE_DEVICES=0,1 SGLANG_DP_SIZE=2 \
bash serve_llada_ui.sh /path/to/LLaDA-UI-checkpoint
Then run one of the packaged requests:
export SGLANG_BASE_URL=http://127.0.0.1:30000/v1
export SGLANG_MODEL=LLaDA-UI
python3 inference/sglang_client.py --example mobile
python3 inference/sglang_client.py --example desktop
python3 inference/sglang_client.py --example web
Output Format
For GUI navigation, a typical response contains tagged reasoning followed by a structured action:
<think>Reasoning grounded in the current GUI state.</think>
<action>Click(box=(x,y))</action>
Here, x and y are integer-normalized to [0, 999]. Despite the field name
box, point-based actions contain one interaction coordinate rather than a
rectangular bounding box.
For GUI grounding, the model directly returns [x,y]. It returns [-1,-1]
when the requested target is infeasible or unrelated to the screenshot.
Citation
@misc{gu2026lladauibringingblockwisediffusion,
title={LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents},
author={Zhangxuan Gu and Haoxing Chen and Qi Qin and Yi Xin and Kai Gan and Lin Liu and Long Cui and Xiaomei Wang and Beitong Zhou and Yunzhu Zhang and Zhengwen Zeng and Changlong Gao and Weizhi Chen and Rongchao Zhang and Haoyuan Wu and Shuheng Shen and Changhua Meng and Weiqiang Wang and Jianguo Li and Zhenzhong Lan},
year={2026},
eprint={2609.13287},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2609.13287},
}