# Computer Use API: How the Screenshot-Action Loop Works Across Anthropic, OpenAI and Google

> How the vendor computer use API loop works: screenshot in, structured action out, harness executes, new screenshot back. Tool names, coordinate systems, step caps, confirmation gates and what to log, for Anthropic, OpenAI and Google.

Guide: Browser agents and agentic browsing in production — part 1
Published: 2026-11-12 · Updated: 2026-11-14 · 1668 words · ~2218 tokens (estimate)
Canonical: https://changegamer.ai/articles/computer-use-api-loop
JSON: https://changegamer.ai/api/articles/computer-use-api-loop.json
Access: free — editorial guide, not part of the licensed corpus
Pillar: https://changegamer.ai/articles/agentic-browsing-in-production.md

## In short

- A computer use API runs a loop in which the harness sends a screenshot, the model returns one or more structured actions, the harness executes them, and a new screenshot goes back until the model stops requesting actions.
- As of 12 November 2026 Anthropic's computer use API takes coordinates in screenshot pixels and Google Gemini uses a normalized 1000x1000 grid, while the OpenAI guide says to use screenshot coordinates and to map them back to the environment's coordinate space if you downscale, but defines no origin or unit grid, so check it before building.
- The computer use loop needs a stop condition that you choose: Anthropic's sample code caps the loop at max_iterations = 10, which is a sample value and not a recommendation.
- Each vendor expects a human confirmation path for consequential actions: OpenAI raises pending safety checks, Gemini returns a safety_decision that can require confirmation, and Anthropic advises human confirmation for real-world decisions.
- ChangeGamer runs no computer-use agent: its only fetcher of third-party sites, the free agent-readiness grader, sends ordinary HTTP GET requests and never renders a page, takes a screenshot or clicks.

---

The computer use API is a loop: your harness sends the model a screenshot, the model answers with one or more structured actions, for example a click or keystroke, your code performs them, and the next screenshot goes back, repeating until the model asks for nothing more. All three vendors work this way as of 12 November 2026. This article covers the call and response shapes, the coordinate traps, stopping, confirmation and logging. For the wider decisions, start at the [agentic browsing pillar](/articles/agentic-browsing-in-production); the [computer use and browser automation reference](/resources/computer-use-browser-automation) holds the contract-level detail. Where it differs from this article (Anthropic coordinate scaling, the OpenAI shutdown date), this article's dated vendor pages are the newer source; the reference refresh is still open.

Vendor details below come from each vendor's own documentation, read on 12 November 2026: [Anthropic](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool), [OpenAI](https://developers.openai.com/api/docs/guides/tools-computer-use) and [Google](https://aistudio.google.com/docs/computer-use). Model names rotate, so check those pages for current models.

## How does the computer use API loop work?

The loop works as four repeating steps, and the model never touches the screen itself: your harness does.

1. **Observe.** The harness captures a screenshot of the browser or desktop and sends it with the goal.
2. **Decide.** The model returns one or more structured actions per turn: a click position, text to type or a scroll.
3. **Act.** The harness executes that action on a real display, or refuses to.
4. **Report.** The harness captures a new screenshot and returns it as the result of the action.

Because the harness is your code, every safeguard in this article is something you build. The vendor supplies the model and the action vocabulary; you supply the sandbox, the cap, the gates and the log.

## What do Anthropic, OpenAI and Google each expose?

Each vendor exposes one named tool with its own action set and message shape, and the differences matter when you write the dispatcher.

| Vendor | Tool | Action set (per vendor doc, 12 November 2026) | Call shape |
|---|---|---|---|
| Anthropic | `computer_toolset_20260801`, no beta header | 17 actions: screenshot, zoom, left_click, right_click, middle_click, double_click, triple_click, left_click_drag, mouse_move, left_mouse_down, left_mouse_up, cursor_position, scroll, type, key, hold_key, wait | A `tool_use` block with `"toolset_name": "computer"` |
| OpenAI | `computer`, in the Responses API | click, double_click, drag, move, scroll, keypress, type, wait, screenshot | A `computer_call` containing an ordered `actions` array |
| Google | `computer_use` with an `environment` of browser, mobile or desktop | Action names in the doc's examples include click, double_click, type, wait, navigate, go_back, go_forward, open_app and long_press; the page gives no full list | A `function_call`, answered by a `function_result` |

Anthropic's earlier tool versions still exist: `computer_20251124` needs the beta header `computer-use-2025-11-24`, and `computer_20250124` needs `computer-use-2025-01-24`. Anthropic's page says, as of 12 November 2026, that a `computer_20251124` integration must be upgraded before you move it to a model that does not accept that version, such as a Claude 5.5 model, so match the tool version to the model.

OpenAI's page describes three integration options: the built-in `computer` tool, a code-execution integration where the model gets a function tool that accepts a script, and a custom harness over ordinary function calling or remote MCP tools. Its guide points older `computer-use-preview` integrations to a migration guide.

### What does a turn look like on the wire?

A turn differs mainly in what you return. With OpenAI you execute the actions in order, capture the screen, and return a `computer_call_output` whose `call_id` matches, continuing with `previous_response_id`. The screenshot arrives as a `computer_screenshot` PNG data URL, and OpenAI advises `detail: "original"` to preserve resolution. With Gemini the new screenshot goes back inside the `function_result`. With Anthropic it goes back as a `tool_result`.

## Which coordinate system does each API use?

Anthropic uses screenshot pixels and Gemini a normalized 1000x1000 grid, and mixing the two up is a common source of clicks that land in the wrong place (inference).

- **Anthropic: screenshot pixels.** The documentation says coordinates are in screenshot pixels. If you scale screenshots before sending them, scale the returned coordinates back up before clicking. On a macOS retina display, downscale by 2x or halve the coordinates.
- **Google: a normalized 1000x1000 grid.** The client converts: `x / 1000 * screen_width`, and the same for y. Google's example viewport is 1440x900.
- **OpenAI: screenshot coordinates, unit grid not stated.** The guide says to use screenshot coordinates and to map coordinates back to the environment's coordinate space if you downscale; it gives no origin or unit grid, so check before building. Because one `computer_call` can carry several actions, you may run several steps between screenshots; a wrong early step plausibly affects the later ones (inference), so validate each action before running the batch.

Anthropic's sizing guidance for captured screens (the current toolset takes no declared display size) recommends 1024x768 or 1280x720 for general desktop tasks and 1280x800 or 1366x768 for web applications, and says to avoid resolutions above 1920x1080. Newer models accept up to 2,576 px on the long edge. The API does not downscale for you, so an oversized `tool_result` image fails validation.

```bash
# Gemini-style conversion: normalized to a 1000x1000 grid, converted to pixels on a 1440x900 viewport.
awk -v nx=500 -v ny=250 -v w=1440 -v h=900 'BEGIN { printf "click at %d,%d\n", nx/1000*w, ny/1000*h }'
# click at 720,225
```

## When should the loop stop?

The loop should stop when the model stops asking for actions or when your cap fires, whichever comes first, and the cap is your decision. Anthropic's page says its loop continues until Claude responds without requesting any tools or the maximum iteration limit is reached, and that the limit prevents infinite loops that could cause unexpected API costs. Its sample code uses `max_iterations = 10`. Treat that as a sample value for a demo, not a recommendation: the right number depends on your task, and the vendor docs we read give no figure for it.

OpenAI's guide says to continue until the model stops returning `computer_call` items. Google's documented Gemini loop repeats the screenshot, call and result cycle until the task finishes or ends. An iteration cap alone is not enough. Add a wall-clock limit, and log the reason each run ended (finished, iteration cap, time limit, safety stop, human refusal) so capped runs can be counted. The [guide to timeouts and deadlines for AI agents](/articles/timeouts-and-deadlines-for-ai-agents) covers the deadline side, and [what long runs cost](/articles/browser-agent-cost-and-latency) has its own article in this cluster.

## Where do confirmation gates sit?

Confirmation gates sit between the model's decision and your harness executing it, and each vendor gives you a different hook for them.

| Vendor | Hook (per vendor doc, 12 November 2026) | What your code does |
|---|---|---|
| OpenAI | Pending API safety checks | OpenAI's sample stops on them; add a human approval callback only when your application can handle them |
| Google | `safety_decision` on a call; `require_confirmation` | Prompt the end user; if confirmed, set `safety_acknowledgement` in the `function_result`; halt if blocked |
| Anthropic | A precaution, not a response field | Human confirmation for decisions with meaningful real-world consequences, such as accepting cookies, financial transactions or agreeing to terms of service |

OpenAI also tells integrators to confirm consequential actions and to keep users in control of purchases, data transmission and destructive changes. Google states that your application should always implement safety-decision handling regardless of configured overrides. All three vendors also advise isolation: a dedicated VM or container with minimal privileges (Anthropic), an isolated browser or VM with an allow list of sites and actions (OpenAI), and a sandboxed environment (Google). Anthropic adds that its classifiers scan returned screenshots for prompt injection; page-borne injection is covered separately in this cluster, and the general defense is in [how to defend an AI agent against prompt injection](/articles/prompt-injection-defense-in-depth-for-agents). For sandbox choices, see [how to choose a sandbox](/articles/choosing-a-sandbox-for-ai-agent-code-execution).

## What should you log per iteration?

Log enough per iteration to replay the run and explain any action, and treat the field list below as our inference, not vendor guidance. None of the three docs we read prescribes a log schema.

- **Run id, step number and timestamp.**
- **Observation reference:** a screenshot hash or storage key, plus the display size sent.
- **Model action:** the raw action as returned, before any scaling.
- **Executed action:** the action after coordinate conversion, and whether it ran, was skipped or was refused.
- **Gate outcome:** any safety check or confirmation raised, who answered and when.
- **Usage and latency:** token usage fields from the response, and elapsed time for the model call and the capture.
- **Stop reason,** on the final record.

Keep raw and executed coordinates both: the gap between them exposes scaling bugs immediately. Screenshots can contain personal data, so apply the same redaction and retention rules as the rest of your traces; the [guide to redacting sensitive data from agent traces](/articles/redacting-sensitive-data-from-agent-traces) is the starting point.

## What does ChangeGamer itself run?

ChangeGamer runs no computer-use agent. Its only fetcher of third-party sites, the free agent-readiness grader at /api/grader, sends ordinary HTTP(S) GET requests for a site's robots.txt, llms.txt, sitemap, homepage, /license.xml and a few .well-known files and checks the returned text; it never renders a page, takes a screenshot or clicks. We checked the worker code on 12 November 2026. Everything above is therefore a reading of the vendor documentation, not a report from running these loops in production.

## What comes next in this cluster?

The harness choice now has its own article, [browser-use vs Playwright MCP vs Stagehand](/articles/browser-agent-harness-comparison). The cost side is in [what a browser agent costs per task](/articles/browser-agent-cost-and-latency); other planned articles in this cluster cover page-borne prompt injection and logged-in sessions. Until they are published, the [pillar](/articles/agentic-browsing-in-production) gives each an overview, and the [reference](/resources/computer-use-browser-automation) records the vendor contracts as of its own update date. Re-read the three vendor pages before you build: tool versions, beta headers and supported models changed between the versions listed here, and they will change again.

## Frequently asked questions

### What is the difference between the Anthropic and OpenAI computer use APIs?

As of 12 November 2026, Anthropic returns a tool_use block with screenshot-pixel coordinates, while OpenAI's Responses API computer tool returns a computer_call holding an ordered actions array that the harness executes in order before sending back one screenshot.

### Do you have to resize screenshots for a computer use API?

You may have to resize them, and Anthropic is explicit that its API does not downscale for you: an oversized tool_result image is rejected with a validation error. If you scale screenshots down, scale the model's returned coordinates back up before clicking.

### How do you stop a computer use agent from looping forever?

Set an explicit iteration cap and a wall-clock limit in your own harness, because each vendor loop ends only when the model stops requesting actions. Anthropic's documentation says its cap exists to prevent infinite loops and unexpected API costs, and its sample uses 10.

### Does Gemini computer use need a human to confirm actions?

When the response carries a safety_decision of require_confirmation, in which case your application must prompt the end user and, if they agree, set safety_acknowledgement in the function result. Google says to implement this handling regardless of configured overrides.


---

## The rest of this guide

- [Agentic Browsing in Production: Building, Securing and Preparing for Browser Agents](https://changegamer.ai/articles/agentic-browsing-in-production.md): An operator guide to agentic browsing: the computer-use loop, accessibility-tree vs screenshot harnesses, cost and latency, page-borne prompt injection, logged-in-session risk, Web Bot Auth, WebMCP for site owners, evaluation and a launch sequence.
- [browser-use vs Playwright MCP vs Stagehand: Which Open Browser Harness to Pick](https://changegamer.ai/articles/browser-agent-harness-comparison.md): browser-use vs Playwright MCP vs Stagehand compared as of 13 November 2026: how each exposes the browser to the model, grounding, language, session handling, the security caveats the READMEs state, hosted options and a decision table.
- [Browser Agent Cost per Task: A Formula, the Levers and the Latency Caveat](https://changegamer.ai/articles/browser-agent-cost-and-latency.md): How to compute browser agent cost per task as of 14 November 2026: a formula for screenshots and toolset overhead, an illustrative worked example, Anthropic image-token sizes, the cost levers and why the vendor pages read on 14 November 2026 give no latency figure.
- [Prompt Injection in AI Browser Agents: Five Vectors and the Controls Vendors Document](https://changegamer.ai/articles/prompt-injection-against-browser-agents.md): ChangeGamer's grouping of the browser-specific prompt injection vectors as of 15 November 2026 (hidden page text, user-generated content, CometJacking URLs, screenshot text, seeded inbox emails) and which vendor-documented control addresses each, with a vector-to-control table.
- [How to Isolate a Browser Agent From a Logged-In Session](https://changegamer.ai/articles/logged-in-sessions-and-blast-radius.md): How to choose where a browser agent's session state comes from as of 16 November 2026 (your real profile, a persistent agent profile, a fresh profile, an injected Playwright storageState file, or a scoped API credential) and how to bound what it can reach.
- [Headless Browser for AI Agents: Local, Remote or Managed?](https://changegamer.ai/articles/remote-browsers-and-anti-bot.md): How to choose between a local headless browser and a managed remote browser (Browserbase, Bright Data, Cloudflare Browser Run) or self-hosted Crawlee as of 17 November 2026, how agents connect over CDP, and where anti-bot handling stops.
- [How to Sign Browser Agent Requests With Web Bot Auth](https://changegamer.ai/articles/web-bot-auth-for-browser-agents.md): How a team running a browser agent signs its outbound requests with Web Bot Auth as of 18 November 2026: Ed25519 key, signed key directory, the three headers, the Signature-Agent divergence, Cloudflare Verified status and key rotation.
- [How to Add WebMCP Tools to Your Website](https://changegamer.ai/articles/webmcp-for-site-owners.md): How a site owner adds WebMCP tools as of 19 November 2026: origin trial tokens, the testing flag, document.modelContext.registerTool, annotations, the Permissions Policy tools feature, security risks, feature detection and which flows to expose.
- [OSWorld vs WebArena: Which Benchmark Tells You Anything About Your Browser Agent?](https://changegamer.ai/articles/evaluating-browser-agents.md): OSWorld vs WebArena as read on 20 November 2026: what OSWorld, OSWorld-Verified, OSWorld 2.0, WebArena, VisualWebArena and Online-Mind2Web each measure, how they are scored, what running them involves and what none of them covers.

## Reference resources

- https://changegamer.ai/resources/computer-use-browser-automation.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt
Licensing: https://changegamer.ai/api/pricing.json (offer catalog) · https://changegamer.ai/api/payment.json (payment methods, HTTP 402 flow) · access guide: https://changegamer.ai/resources/access-and-pricing.md
