Skip to content
Optimal Lab

AI Engineering · interactive

Computer-use and browser agents

A computer-use agent sees the screen only as screenshots and acts only through clicks and keystrokes. Your harness runs the loop: screenshot, model, one action, run it, screenshot again. Watch it change an order's address, get blocked by a cookie banner, and meet instructions hidden on the page.

You

gave the task

Agent harness

action 1

Model

1.0k tokens

Browser (VM)

order

screenshot 1 · 1024×768

Acme Shop
Account

Order #4471 · 1 desk lamp

Ship to: 9 Mill Road

Edit address

This is order #4471. There's an “Edit address” button next to the shipping address. → The address form opens.

Scenario

turn 1 of 5
  1. 1

  2. 2

  3. 3

    A tool call with an action and pixel coordinates, or text to type. The model has to locate the button in the image itself.

    {"action": "left_click", "coordinate": [820, 300]}

  4. 4

  5. 5

In production

Tools and technologies

Computer-use models plus a harness that takes screenshots and performs actions in a sandboxed browser or VM.

Not tied to one language — use these names when you search docs or design a system.