AGW-35B

A cross-platform GUI agent trained on synthetic interaction trajectories from AutoGUIWorld

Project · Model Weights

Overview

AGW-35B is a screenshot-based computer-use agent obtained by fine-tuning Qwen3.5-35B-A3B on 79,266 grounded and filtered GUI training examples generated by AutoGUIWorld. Given a task instruction, the current screenshot, and recent interaction history, the model predicts the next executable computer_use action.

AutoGUIWorld expands GUI training coverage without executing every trajectory in the corresponding software environment. A planner specifies atomic actions and their intended visual consequences; an image generator then synthesizes successive GUI states. The training set spans Chrome, Ubuntu, Windows, and macOS, including browser, office, creative, system, and scientific workflows.

AGW-35B is trained as a direct-action, non-thinking agent. Its response contains a short action description followed by one or more structured tool calls. Spatial actions use a resolution-independent 0–999 coordinate grid, which the executor maps to the current screen resolution.

Model at a Glance

Results

The final checkpoint is evaluated with the same task set as the base model. Interactive benchmarks allow up to 100 agent turns per task. OSWorld and Windows Agent Arena retain partial credit; macOSWorld and ScienceBoard report task success.

AGW-35B also improves visual grounding on the 1,581-example English positive-target split of ScreenSpot-Pro.

All values are percentages. The metrics differ across benchmarks and should be interpreted within each benchmark rather than pooled across individual tasks.

Training Data

The AutoGUIWorld training set contains 79,266 examples distributed across four GUI environments:

Each example is derived from a multi-step screenshot trajectory and contains the task, current observation, recent action history, and the next action target. AutoGUIWorld provides spatial grounding for pointing actions and applies transition-level quality control before examples enter training.

Training Configuration

AGW-35B was trained with ms-swift Megatron on 32 GPUs across four nodes. The language model was fine-tuned while the visual encoder was frozen.

Overlength examples were removed rather than truncated. The checkpoint uses the Qwen3.5 MoE architecture with 256 experts and 8 selected experts per token.

Interaction Contract

The model consumes screenshot-centered multi-turn messages and emits direct actions in the following form:

Action: Click the Settings icon.
<tool_call>
{"name":"computer_use","arguments":{"action":"left_click","coordinate":[927,54]}}
</tool_call>

The action space covers mouse movement and clicks, dragging, keyboard input, typing, scrolling, waiting, and task termination. Coordinates are integers on a 0–999 grid. An external agent harness must execute the tool call, capture the next screenshot, and append the resulting turn to the interaction history.

AGW-35B was trained without target-side chain-of-thought. For consistent behavior, disable thinking at inference and preserve the repository's chat template.

Serving with vLLM

Use a recent Qwen3.5-compatible vLLM build. The following configuration matches the 16,384-token SFT context and the tensor-parallel setup used in evaluation:

uv pip install vllm --torch-backend=auto \
  --extra-index-url https://wheels.vllm.ai/nightly

vllm serve YangC777/AGW-35B \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --dtype bfloat16

A minimal single-step request through the OpenAI-compatible endpoint is shown below. In a full agent loop, replace the compact system message with the project harness prompt and append previous screenshot-action turns.

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

with open("screenshot.png", "rb") as f:
    screenshot = base64.b64encode(f.read()).decode("utf-8")

system_prompt = """You are a GUI agent. Given a task and screenshot, output exactly:
Action: <one short imperative>
<tool_call>
{"name":"computer_use","arguments":{"action":<action>, ...}}
</tool_call>
Use integer coordinates on a 0-999 grid. Use terminate when the task is complete."""

response = client.chat.completions.create(
    model="YangC777/AGW-35B",
    messages=[
        {"role": "system", "content": system_prompt},
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{screenshot}"},
                },
                {
                    "type": "text",
                    "text": "# Task Instruction:\nOpen Settings and switch to dark mode.\n\n"
                            "Generate the next move from the screenshot.",
                },
            ],
        },
    ],
    temperature=0.0,
    top_p=1.0,
    max_tokens=1024,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

print(response.choices[0].message.content)

Intended Use

AGW-35B is intended for research and development of screenshot-based GUI agents, including:

• cross-platform desktop and browser task execution;
• action grounding in dense professional interfaces;
• evaluation of synthetic GUI trajectories for agent post-training;
• agent systems that execute model-produced actions inside controlled environments.

The checkpoint is specialized for computer use. It should be paired with an executor, environment reset mechanism, action parser, and benchmark-appropriate safety controls.

Limitations

• The model observes rendered screenshots rather than hidden application or file-system state, so visually ambiguous states can lead to incorrect actions.
• Evaluation uses English task instructions and English interfaces; performance in other languages has not been established.
• Long workflows remain sensitive to early action errors and changes in application state.
• Performance varies by application domain. In the reported evaluation, Chrome scores decrease on both OSWorld and Windows Agent Arena even as aggregate performance improves.
• Training trajectories are generated through AutoGUIWorld and can inherit visual or transition artifacts from the generation pipeline.

Citation

AutoGUIWorld paper on arXiv

@misc{yang2026autoguiworldimagegeneratorsvisual,
      title={AutoGUIWorld: Image Generators as Visual World Models for GUI Agent}, 
      author={Cheng Yang and Yifan Wu and Yutao Huang and Zhaohua Zhang and Beiduo Chen and Muxi Chen and Chenchen Zhao and Hexuan Deng and Haolin Yang and Geyuan Zhu and Sa Zhu and Jianhuan Zhuo and Qiuyong Xiao and Jianhao Ruan and Yiran Peng and Jiayi Zhang and Tian Ye and Xinlei Yu and Tianwen Jiang and Jihong Zhang and Yuyu Luo},
      year={2026},
      eprint={2610.01215},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.01215}, 
}

Acknowledgements

AGW-35B is built on Qwen3.5-35B-A3B. We thank the authors and maintainers of Qwen, OSWorld, Windows Agent Arena, macOSWorld, ScienceBoard, and ScreenSpot-Pro.