ChangeGamer

Computer Use API: How the Screenshot-Action Loop Works Across Anthropic, OpenAI and Google

Part 1 of Browser agents and agentic browsing in production · 1,668 words · ~8 min read · published · updated · Markdown variant

How the vendor computer use API loop works: screenshot in, structured action out, harness executes, new screenshot back. Tool names, coordinate systems, step caps, confirmation gates and what to log, for Anthropic, OpenAI and Google.

In short

  • A computer use API runs a loop in which the harness sends a screenshot, the model returns one or more structured actions, the harness executes them, and a new screenshot goes back until the model stops requesting actions.
  • As of 12 November 2026 Anthropic's computer use API takes coordinates in screenshot pixels and Google Gemini uses a normalized 1000x1000 grid, while the OpenAI guide says to use screenshot coordinates and to map them back to the environment's coordinate space if you downscale, but defines no origin or unit grid, so check it before building.
  • The computer use loop needs a stop condition that you choose: Anthropic's sample code caps the loop at max_iterations = 10, which is a sample value and not a recommendation.
  • Each vendor expects a human confirmation path for consequential actions: OpenAI raises pending safety checks, Gemini returns a safety_decision that can require confirmation, and Anthropic advises human confirmation for real-world decisions.
  • ChangeGamer runs no computer-use agent: its only fetcher of third-party sites, the free agent-readiness grader, sends ordinary HTTP GET requests and never renders a page, takes a screenshot or clicks.

Part of the Agentic Browsing in Production: Building, Securing and Preparing for Browser Agents guide.


The computer use API is a loop: your harness sends the model a screenshot, the model answers with one or more structured actions, for example a click or keystroke, your code performs them, and the next screenshot goes back, repeating until the model asks for nothing more. All three vendors work this way as of 12 November 2026. This article covers the call and response shapes, the coordinate traps, stopping, confirmation and logging. For the wider decisions, start at the agentic browsing pillar; the computer use and browser automation reference holds the contract-level detail. Where it differs from this article (Anthropic coordinate scaling, the OpenAI shutdown date), this article's dated vendor pages are the newer source; the reference refresh is still open.

Vendor details below come from each vendor's own documentation, read on 12 November 2026: Anthropic, OpenAI and Google. Model names rotate, so check those pages for current models.

How does the computer use API loop work?

The loop works as four repeating steps, and the model never touches the screen itself: your harness does.

  1. Observe. The harness captures a screenshot of the browser or desktop and sends it with the goal.
  2. Decide. The model returns one or more structured actions per turn: a click position, text to type or a scroll.
  3. Act. The harness executes that action on a real display, or refuses to.
  4. Report. The harness captures a new screenshot and returns it as the result of the action.

Because the harness is your code, every safeguard in this article is something you build. The vendor supplies the model and the action vocabulary; you supply the sandbox, the cap, the gates and the log.

What do Anthropic, OpenAI and Google each expose?

Each vendor exposes one named tool with its own action set and message shape, and the differences matter when you write the dispatcher.

Vendor Tool Action set (per vendor doc, 12 November 2026) Call shape
Anthropic computer_toolset_20260801, no beta header 17 actions: screenshot, zoom, left_click, right_click, middle_click, double_click, triple_click, left_click_drag, mouse_move, left_mouse_down, left_mouse_up, cursor_position, scroll, type, key, hold_key, wait A tool_use block with "toolset_name": "computer"
OpenAI computer, in the Responses API click, double_click, drag, move, scroll, keypress, type, wait, screenshot A computer_call containing an ordered actions array
Google computer_use with an environment of browser, mobile or desktop Action names in the doc's examples include click, double_click, type, wait, navigate, go_back, go_forward, open_app and long_press; the page gives no full list A function_call, answered by a function_result

Anthropic's earlier tool versions still exist: computer_20251124 needs the beta header computer-use-2025-11-24, and computer_20250124 needs computer-use-2025-01-24. Anthropic's page says, as of 12 November 2026, that a computer_20251124 integration must be upgraded before you move it to a model that does not accept that version, such as a Claude 5.5 model, so match the tool version to the model.

OpenAI's page describes three integration options: the built-in computer tool, a code-execution integration where the model gets a function tool that accepts a script, and a custom harness over ordinary function calling or remote MCP tools. Its guide points older computer-use-preview integrations to a migration guide.

What does a turn look like on the wire?

A turn differs mainly in what you return. With OpenAI you execute the actions in order, capture the screen, and return a computer_call_output whose call_id matches, continuing with previous_response_id. The screenshot arrives as a computer_screenshot PNG data URL, and OpenAI advises detail: "original" to preserve resolution. With Gemini the new screenshot goes back inside the function_result. With Anthropic it goes back as a tool_result.

Which coordinate system does each API use?

Anthropic uses screenshot pixels and Gemini a normalized 1000x1000 grid, and mixing the two up is a common source of clicks that land in the wrong place (inference).

Anthropic's sizing guidance for captured screens (the current toolset takes no declared display size) recommends 1024x768 or 1280x720 for general desktop tasks and 1280x800 or 1366x768 for web applications, and says to avoid resolutions above 1920x1080. Newer models accept up to 2,576 px on the long edge. The API does not downscale for you, so an oversized tool_result image fails validation.

# Gemini-style conversion: normalized to a 1000x1000 grid, converted to pixels on a 1440x900 viewport.
awk -v nx=500 -v ny=250 -v w=1440 -v h=900 'BEGIN { printf "click at %d,%d\n", nx/1000*w, ny/1000*h }'
# click at 720,225

When should the loop stop?

The loop should stop when the model stops asking for actions or when your cap fires, whichever comes first, and the cap is your decision. Anthropic's page says its loop continues until Claude responds without requesting any tools or the maximum iteration limit is reached, and that the limit prevents infinite loops that could cause unexpected API costs. Its sample code uses max_iterations = 10. Treat that as a sample value for a demo, not a recommendation: the right number depends on your task, and the vendor docs we read give no figure for it.

OpenAI's guide says to continue until the model stops returning computer_call items. Google's documented Gemini loop repeats the screenshot, call and result cycle until the task finishes or ends. An iteration cap alone is not enough. Add a wall-clock limit, and log the reason each run ended (finished, iteration cap, time limit, safety stop, human refusal) so capped runs can be counted. The guide to timeouts and deadlines for AI agents covers the deadline side, and what long runs cost has its own article in this cluster.

Where do confirmation gates sit?

Confirmation gates sit between the model's decision and your harness executing it, and each vendor gives you a different hook for them.

Vendor Hook (per vendor doc, 12 November 2026) What your code does
OpenAI Pending API safety checks OpenAI's sample stops on them; add a human approval callback only when your application can handle them
Google safety_decision on a call; require_confirmation Prompt the end user; if confirmed, set safety_acknowledgement in the function_result; halt if blocked
Anthropic A precaution, not a response field Human confirmation for decisions with meaningful real-world consequences, such as accepting cookies, financial transactions or agreeing to terms of service

OpenAI also tells integrators to confirm consequential actions and to keep users in control of purchases, data transmission and destructive changes. Google states that your application should always implement safety-decision handling regardless of configured overrides. All three vendors also advise isolation: a dedicated VM or container with minimal privileges (Anthropic), an isolated browser or VM with an allow list of sites and actions (OpenAI), and a sandboxed environment (Google). Anthropic adds that its classifiers scan returned screenshots for prompt injection; page-borne injection is covered separately in this cluster, and the general defense is in how to defend an AI agent against prompt injection. For sandbox choices, see how to choose a sandbox.

What should you log per iteration?

Log enough per iteration to replay the run and explain any action, and treat the field list below as our inference, not vendor guidance. None of the three docs we read prescribes a log schema.

Keep raw and executed coordinates both: the gap between them exposes scaling bugs immediately. Screenshots can contain personal data, so apply the same redaction and retention rules as the rest of your traces; the guide to redacting sensitive data from agent traces is the starting point.

What does ChangeGamer itself run?

ChangeGamer runs no computer-use agent. Its only fetcher of third-party sites, the free agent-readiness grader at /api/grader, sends ordinary HTTP(S) GET requests for a site's robots.txt, llms.txt, sitemap, homepage, /license.xml and a few .well-known files and checks the returned text; it never renders a page, takes a screenshot or clicks. We checked the worker code on 12 November 2026. Everything above is therefore a reading of the vendor documentation, not a report from running these loops in production.

What comes next in this cluster?

The harness choice now has its own article, browser-use vs Playwright MCP vs Stagehand. The cost side is in what a browser agent costs per task; other planned articles in this cluster cover page-borne prompt injection and logged-in sessions. Until they are published, the pillar gives each an overview, and the reference records the vendor contracts as of its own update date. Re-read the three vendor pages before you build: tool versions, beta headers and supported models changed between the versions listed here, and they will change again.

Frequently asked questions

What is the difference between the Anthropic and OpenAI computer use APIs?
As of 12 November 2026, Anthropic returns a tool_use block with screenshot-pixel coordinates, while OpenAI's Responses API computer tool returns a computer_call holding an ordered actions array that the harness executes in order before sending back one screenshot.
Do you have to resize screenshots for a computer use API?
You may have to resize them, and Anthropic is explicit that its API does not downscale for you: an oversized tool_result image is rejected with a validation error. If you scale screenshots down, scale the model's returned coordinates back up before clicking.
How do you stop a computer use agent from looping forever?
Set an explicit iteration cap and a wall-clock limit in your own harness, because each vendor loop ends only when the model stops requesting actions. Anthropic's documentation says its cap exists to prevent infinite loops and unexpected API costs, and its sample uses 10.
Does Gemini computer use need a human to confirm actions?
When the response carries a safety_decision of require_confirmation, in which case your application must prompt the end user and, if they agree, set safety_acknowledgement in the function result. Google says to implement this handling regardless of configured overrides.

#computer-use #browser-agents #anthropic #openai #gemini #agent-loop #safety-checks

Put this corpus inside your own agents

This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.

Agents: this page as Markdown · JSON · offers at /api/pricing.json · payment methods at /api/payment.json · single-resource access via HTTP 402 (how that works)