Computer-Use Agents: The State of Clicking Things

A computer-use agent takes a screenshot, decides where to click, clicks there, and takes another screenshot. Everything difficult about it follows from the fact that it must do this thirty or forty times in a row without getting one of them wrong.

This page is deliberately dated: the capability is moving, and any success rate written here would be stale within months. What does not change is the arithmetic underneath, which is what lets you read a published number and know what it implies for your workflow.

The loop, concretely

It is the same loop as any other agent with two changes: the observation is an image, and the action space is a mouse and a keyboard rather than a set of domain functions.

Two implementation details do a great deal of work. Screenshots are usually downscaled to a fixed resolution before being sent, and the model’s coordinates are then scaled back up — so the agent’s spatial precision is bounded by that intermediate resolution, and getting the transform wrong produces clicks that are consistently offset by a few percent. And most implementations draw no cursor, so the model cannot see where the pointer currently is; state it believes about focus and hover is inferred, not observed.

The token arithmetic

Images are expensive and they accumulate. Anthropic’s vision documentation gives a working approximation of tokens ≈ (width × height) / 750, which for a 1280×800 screenshot is about 1,365 tokens. Now put that in a loop where every request carries the whole history:

one 1280x800 screenshot        ~1,365 tokens

keeping every screenshot, 40 steps:
  request i carries i images
  total image tokens = 1,365 * (1+2+...+40)
                     = 1,365 * 820
                     = 1,119,300 tokens  per single 40-step task

keeping only the last 3 screenshots:
  total image tokens = 1,365 * 3 * 40
                     = 163,800 tokens    (~7x less)

Which is why every serious implementation prunes old images, replacing them with a text stub — [screenshot at step 12: settings page, elided] — so the fact that the step happened survives while the pixels do not. The pruning is not an optimisation, it is what makes a forty-step run affordable at all. It also has a cost: the agent loses the ability to compare the current screen with one from twenty steps ago, which is exactly what it needs when verifying that a change took effect. Keeping a small number of explicitly pinned screenshots — the starting state, the last verified checkpoint — buys most of that back.

Why long tasks fail

This is the arithmetic that explains every benchmark result you will read, and it is worth internalising because it also tells you which tasks are feasible. If each step succeeds independently with probability p, an n-step task with no recovery succeeds with probability p^n:

per-step p    5 steps   10 steps   20 steps   40 steps
   0.90         0.59       0.35       0.12       0.015
   0.95         0.77       0.60       0.36       0.13
   0.97         0.86       0.74       0.54       0.30
   0.99         0.95       0.90       0.82       0.67

A model that gets 97% of individual clicks right — which sounds excellent, and is — completes a forty-step workflow about 30% of the time. That single table reconciles the two things people say about computer use: that the demos are impressive, and that it does not work. Both are true, and they are the same number viewed at different values of n.

Two consequences for design. First, shortening the task raises success superlinearly: getting a workflow from 20 steps to 10 by deep-linking past the navigation is worth more than any prompt engineering. Second, the assumption of no recovery is the one to attack — an agent that can detect a wrong state and undo it converts an independent-failure model into something much better, which is why verification steps (“screenshot and confirm the dialog closed”) earn their cost even though they add to n.

Where the real numbers live

Published, maintained, and moving too fast to quote here:

• OSWorld — real desktop tasks across applications on a real OS, with execution-based verification rather than a judge. The most cited general computer-use benchmark, and notable for reporting a human baseline alongside the agent scores, which is the comparison that matters.
• WebArena and WebVoyager — web tasks on self-hosted and live sites respectively. The self-hosted design exists because live sites make results irreproducible, which is worth remembering when reading any demo against a real website.
• Windows Agent Arena — the same idea in a Windows environment, with parallelised evaluation.

Read them from the leaderboards rather than from articles, check the date on any figure, and check whether the number is a single-attempt success or a best-of-k. The distinction between those two is usually the difference between the headline and the experience.

What actually goes wrong

• Modal interruptions. Cookie banners, update prompts, “rate this app”, session expiry. Each one is an unplanned state the agent must recognise and dismiss, and each is a step at which the compounding table applies.
• Focus loss. The agent types a password into a window that stopped being focused two steps ago. Nothing on screen necessarily indicates this.
• Scroll containers. The target is visible in a screenshot but inside a nested scroll region, so the coordinates are right and the click lands on an overlay.
• Timing. Screenshot taken mid-animation, mid-render, or before an async update. The model reasons correctly about a screen that no longer exists by the time the click lands.
• Injection from the screen. The observation is attacker-controllable. A web page can contain text addressed to the agent — and the agent is reading the page. This is the reason computer use belongs behind a real sandbox with a scoped profile and no access to credentials it does not need.
• CAPTCHAs and bot detection. Not a bug to be solved. A workflow that hits one has told you it is not a workflow for an agent.

When to use it at all

Computer use is the interface of last resort, and that is a compliment — it exists precisely for systems that have no other interface. The decision procedure is short:

• Is there an API? Use it. An API call is one step with a status code, not eight steps with a screenshot each.
• Is there a stable DOM? Then a browser-automation tool driving accessibility-tree selectors is far more reliable than pixel coordinates, and roughly free by comparison.
• Is the flow short and does it repeat? Then a recorded deterministic script beats an agent, with the agent reserved for the exception path when the script breaks.
• Is it a legacy desktop application with no interface at all? This is the case computer use was built for, and it is genuinely valuable there — with a step budget, a sandbox, and a human gate on anything irreversible.