PhoneUIAnchor-829M
PhoneUIAnchor-829M is a vision-language model for locating elements in mobile and web interfaces. Given a screenshot and a natural-language description, it returns the normalized center point of the target element. It can be used with GUI agents, test automation, accessibility tools, and similar visual workflows.
This repository contains a complete, standalone checkpoint that can be loaded directly with Transformers without additional model files.
Capabilities
- Intent grounding: locate the control required to complete an action.
- Description grounding: locate an element from visible or semantic traits.
- Function grounding: locate a control from a detailed functionality description.
- Cross-resolution output: return normalized 0-999 coordinates that can be mapped to any screenshot size.
- Local deployment: run from a single 829M-parameter model without an external inference API.
Technical Profile
| Property | Value |
|---|---|
| Architecture | Florence-2-large |
| Task | GUI element grounding |
| Parameters | 829M |
| Output contract | <loc_x>,<loc_y> |
| Weight format | Safetensors |
| Recommended precision | BF16 |
| Tested stack | Python 3.10, PyTorch 2.4.1, CUDA 12.1 |
Installation
pip install -r requirements.txt
Usage
The packaged Florence-2 implementation uses custom model code. Review the
included Python files and load with trust_remote_code=True.
import re
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor
model_id = "lumimate/PhoneUIAnchor-829M"
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16,
attn_implementation="sdpa",
).cuda().eval()
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
image = Image.open("ui_screenshot.png").convert("RGB")
prompt = (
'Where is the "Settings" element? '
'(Output the center coordinates of the target)'
)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(
"cuda", dtype=torch.bfloat16
)
with torch.inference_mode():
output_ids = model.generate(
**inputs,
do_sample=False,
max_new_tokens=16,
)
text = processor.tokenizer.batch_decode(
output_ids, skip_special_tokens=False
)[0]
match = re.search(r"<loc_(\d+)>,<loc_(\d+)>", text)
point = tuple(map(int, match.groups())) if match else None
x_px = point[0] / 999 * image.width
y_px = point[1] / 999 * image.height
print({"normalized": point, "pixels": (x_px, y_px)})
A command-line implementation is included in inference_example.py.
Intended Use
PhoneUIAnchor-829M is intended for research and product prototyping in GUI perception, agentic interaction, automated testing, and assistive interfaces. Predictions should be validated before high-impact or irreversible automation.
Limitations
- Development data emphasizes mobile user interfaces and may not cover every desktop, game, embedded, or highly customized UI style.
- Coordinate quality can degrade on very small, occluded, or visually ambiguous controls.
Architecture and Licensing
PhoneUIAnchor-829M uses the Florence-2 architecture and is distributed under the
terms provided in LICENSE.