Model Card for Proxy Lite

Model Description

• Developed by: Convergence AI
• Model type: 3B Vision-Language Model
• Agent type: Web-browsing Agent
• License: CC-BY-NC-4.0
• Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct
• Running the agent

Running Proxy on the web

https://github.com/convergence-ai/proxy-lite to run Proxy lite on a browser

git clone https://github.com/convergence-ai/proxy-lite.git
make proxy
proxy "Find some markets near Kings Cross and tell me their ratings."

Uses

Proxy Lite is designed and trained to complete automated tasks in a web browser.

Full code for running the model is available in the github repository.

This includes a CLI tool for running the model, as well as a streamlit app.

You can use this endpoint for small-scale testing.

Direct Use

We recommend hosting your own endpoint with vLLM, you can use the following command:

vllm serve convergence-ai/proxy-lite-3b \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --port 8008 \

The tool arguments are very important for parsing the tool calls from the model appropriately.

Important: Qwen-2.5-VL Support in transformers is not yet available in the latest release so be sure to install from source.

Message History

When it comes to using and prompting Proxy Lite, please refer to the repository for more information, but the model expects a message history of the form:

message_history = [
    {
        "role": "system", 
        "content": "You are Proxy Lite...", # Full system prompt in src/proxy_lite/agents/proxy_lite_agent.py
    }, # System prompt
    {
        "role": "user", 
        "content": "Find some markets near Kings Cross and tell me their ratings.",
    }, # Set the task
    {
        "role": "user", 
        "content": [
            {"type": "image_url", "image_url": {base64_encoded_screenshot} },
            {"type": "text", "text": "URL: https://www.google.com/ \n- [0] <a>About</a> \n- [1] <a>Store</a>...."}
        ] # This is the observation from the environment
    },
]

This would then build up the message history, alternating between the assistant (who takes the action) and the user (who provides the observation).

Context-Window Management: When making calls to the model, all the observations other than the current one are discarded in order to reduce the large number of image tokens required. Since the model responses include reflection on the observations and are all included in the message history, the model is still aware of the entire history when planning new actions.

Tools

You should also pass the Tools that the model has access to, these will define the action space available to the model. You can do this with transformers:

from qwen_vl_utils import process_vision_info
from transformers import AutoProcessor

from proxy_lite.tools import ReturnValueTool, BrowserTool
from proxy_lite.serializer import OpenAICompatableSerializer

processor = AutoProcessor.from_pretrained("convergence-ai/proxy-lite-3b")
tools = OpenAICompatableSerializer().serialize_tools([ReturnValueTool(), BrowserTool(session=None)])

templated_messages = processor.apply_chat_template(
    message_history, tokenize=False, add_generation_prompt=True, tools=tools
)

image_inputs, video_inputs = process_vision_info(message_history)

batch = processor(
    text=[templated_messages],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)

Or you can send to the endpoint directly, which will handle the formatting:

from openai import OpenAI

client = OpenAI(base_url="http://convergence-ai-demo-api.hf.space/v1")

response = client.chat.completions.create(
    model="convergence-ai/proxy-lite-3b",
    messages=message_history,
    tools=tools,
    tool_choice="auto",
)

Evaluation

Proxy Lite scored 72.4% on the WebVoyager benchmark, placing it 1st out of all available open-weights models.

A breakdown of the results by website is shown below:

Out-of-Scope Use

Proxy Lite is specifically designed to automate routine tasks within a web browser environment. However, it should not be used for:

• High-Stakes or Safety-Critical Applications:
  Avoid using Proxy Lite for tasks such as financial transactions, healthcare operations, legal decision-making, or emergency responses, where any error could lead to serious harm or significant financial loss.

• Unauthorized or Invasive Data Extraction:
  Automated scraping or extraction of data from websites should only be performed with explicit permission. Proxy Lite should not be used to bypass websites' terms of service, copyright restrictions, or privacy policies.

• Interactions with Malicious or Unverified Websites:
  Using the model to navigate or interact with suspicious or untrusted websites may expose the system to security threats such as malware, phishing attacks, or other forms of cyber exploitation.

• Compliance-Regulated or Legally Sensitive Actions:
  Tasks that require adherence to strict legal or regulatory standards (e.g., processing personal data or sensitive information) should employ additional safeguards beyond what the model provides.

Citation

BibTeX:

@article{proxy-lite,
  title={Proxy Lite - A Mini, Open-weights, Autonomous Assistant},
  author={Convergence AI},
  year={2025}
}