Screenshot Understanding for UI Automation

Driving a user interface from a screenshot is the most demanding coordinate task in this cluster, because being approximately right is worthless. A click 30 pixels off does not land 30 pixels from the button; it lands on a different control, and the run continues confidently into a state nobody planned for.

The task, stated precisely

Given a screenshot and an instruction, return a coordinate to click. It decomposes into three subproblems that fail independently and are worth instrumenting separately: identifying the correct target (“the Save button, not Save As”), localising it in the image, and converting that location into a coordinate in the coordinate system your automation driver expects.

The third is not a model problem at all. It is arithmetic, and it is where most production error comes from.

The bug that eats most of your accuracy

The model never saw your screenshot. It saw a resized version of it, in whatever dimensions the provider’s preprocessing produced, and its coordinates refer to that. Providers additionally differ in what space they report in — some model families are documented as emitting coordinates normalised to a fixed 0–1000 grid regardless of the actual image size, others report pixels in the resized image, others expect you to state the convention in the prompt.

screenshot            2560 x 1440   (retina, 2x device pixel ratio)
provider downscales   1092 x 614    (what the model actually sees)
model returns         (546, 307)    center of its view

wrong: click at (546, 307)
       -> lands at 21% across, 21% down. Nowhere near.

right: scale back to the source image
  sx = 2560 / 1092 = 2.344
  sy = 1440 / 614  = 2.345
  (546 * 2.344, 307 * 2.345) = (1280, 720)   the true centre

then, if the driver works in logical points, not device pixels:
  (1280 / 2, 720 / 2) = (640, 360)

Three coordinate spaces — model view, image pixels, logical screen points — and a factor-of-two device pixel ratio waiting in the third. A pipeline that gets the first transform right and forgets the second produces clicks that are consistently at half the correct offset, which looks exactly like “the model is bad at coordinates” and is not.

Write the transform once, in one function, with the source dimensions passed in explicitly rather than inferred. Then verify it with a synthetic case: a screenshot with a single red square at known coordinates. If that does not come back correct, nothing else you measure means anything.

Two related traps sit next to this one. Scroll position: the model sees a viewport, and coordinates it returns are viewport-relative, while your driver may expect document-relative ones. Any page taller than the window will produce clicks that are correct near the top and increasingly wrong as you scroll, which reads as a mysterious degradation over the course of a run. And multiple displays: a full-desktop capture across two monitors gives coordinates in a combined space with an origin that may not be where you assume, and negative x values are entirely normal when the secondary display sits to the left. Both are arithmetic bugs with visual symptoms, which is exactly the kind that gets misattributed to the model.

Not asking for coordinates at all

The most effective mitigation is to remove the localisation problem from the model’s job. Two approaches:

• Set-of-mark prompting (Yang et al., 2023). Overlay numbered boxes on the candidate elements before sending the screenshot, and ask the model which number to click. It now answers with an integer instead of a coordinate pair, and an integer is either right or wrong rather than approximately right. The candidates come from wherever you can get them — an accessibility tree, the DOM, or a detector.
• Structured tree first, pixels second. In a browser, the DOM already knows where everything is. Send the accessibility tree as text, have the model name the element, and get exact coordinates from the platform. Reserve the screenshot for what the tree cannot express — canvas content, an image-only control, or checking that a dialog is genuinely visible rather than merely present.

Both convert a continuous estimation problem into a discrete selection problem. Where you have any source of element candidates at all, this is worth more than any model upgrade.

The second approach has an underrated side effect: the accessibility tree tells you things a screenshot cannot. Whether a control is disabled, what its accessible name is, whether a dialog is modal, whether an element is present but scrolled out of view. A model looking at pixels has to infer “disabled” from a shade of grey, which is exactly the kind of judgement it gets wrong confidently. If you are automating a browser and using screenshots as the primary channel, you are usually working harder than necessary.

A harness you can run

Ground truth for this task is free if you generate it, which is the part most people skip.

# Playwright gives you both the screenshot and the truth
page.goto(url)
page.screenshot(path="shot.png")

truth = page.eval_on_selector_all(
    "button, a, input, [role=button]",
    """els => els.map(e => {
        const r = e.getBoundingClientRect();
        return {label: (e.innerText || e.ariaLabel || "").trim(),
                x: r.x, y: r.y, w: r.width, h: r.height};
    }).filter(e => e.label && e.w > 0)"""
)

# ask the model for a click point for each label, then:
def hit(pred, box):
    return (box["x"] <= pred[0] <= box["x"] + box["w"] and
            box["y"] <= pred[1] <= box["y"] + box["h"])

Report hit rate, not mean pixel distance — distance is the wrong metric because being inside a large button by 2 pixels and by 200 pixels are equally successful. Then break the result down by target size, because the aggregate hides everything: a 40-pixel-tall primary button and a 12-pixel close icon are different tasks, and only one of them is likely to be your problem. Track refusals and malformed outputs as failures too.

Run the same harness against a set-of-mark variant and against the accessibility-tree route on the same pages, and you have the only comparison that matters for your application: not which model is best at coordinates, but which of three architectures clicks the right thing most often on the interfaces you actually automate. That result tends to be stable across model versions in a way that a raw coordinate score is not, which makes it worth the afternoon.

Production concerns nobody mentions

• Screenshots are expensive tokens. A full-resolution desktop capture per step, in a loop of thirty steps, is a large bill and a large context. Crop to the active window, downscale to the provider’s ceiling yourself, and drop older screenshots from the history rather than accumulating them.
• Verify after acting, not before. The reliable loop is act, screenshot, check that the expected change occurred, and recover if not. Without the check, one bad click cascades silently through everything after it.
• Redact before sending. A screenshot of a real session contains whatever was on the screen — tokens, personal data, other people’s messages. This is a data-handling decision that belongs in the design, not in a retro.
• Nondeterminism is in the UI too. Animations, lazy-loaded content and A/B tests mean two captures of the same page are not the same image. Wait on a condition, not on a timer, before capturing.