Computer use: when the interface is the API

Some work has no API. The vendor portal, the legacy admin panel, the SaaS tool whose export lives behind four clicks — for all of it, the interface is the interface. Computer use closes that gap: Claude looks at screenshots, reasons about what it sees, and issues clicks, keystrokes, and scrolls — the same perception–action loop a human runs, automated. It is the most capable and the most dangerous tool in this course, and this lesson teaches both halves.

The loop, mechanically

Computer use is ordinary tool use with a twist: the "tool" is a virtual display you control. Claude receives a screenshot, decides an action, your code executes it against the environment, and the next screenshot shows the result:

Concretely, you declare the Anthropic-defined computer tool, run a desktop or browser in a sandbox (a VM or container you provision), and your tool handler maps Claude's actions — screenshot, left_click at coordinates, type, key, scroll — onto that environment. Modern Claude models return pixel-accurate coordinates, and the operational sweet spot is a 1080p-class display: enough fidelity to read real UIs, without paying peak image-token prices for every frame. Everything you learned in the tools lesson applies unchanged — results return as blocks, errors are data, the loop runs until the model stops acting.

What makes it work in practice is less the loop than the briefing. Tell Claude what application it is in, what the task's definition of done is, and what it must not touch — the same complete-specification principle from the prompting lesson, applied to an agent that can click anything. And expect iteration: UIs have modals, load states, and surprises, so a computer-use task is a long agentic run with all the context-management needs the next lesson covers.

Claude in Chrome: the packaged version

You met Claude in Chrome in the surfaces lesson; place it now in this frame. It is the same perception–action capability, shipped as a product — running in your real browser with your logged-in sessions, under per-site permissions you grant, with confirmation gates on sensitive actions built in. The decision rule between the two: Chrome for your own recurring browser work; the computer-use API for automation you ship — your infrastructure, your sandbox, your guardrails.

The safety model is not optional

Computer use concentrates every risk this course has mentioned, because the agent's "tool surface" is an entire computer:

• Sandbox everything. A dedicated VM or container with the minimum installed software, minimum network egress, and no ambient credentials. The blast radius of a wrong click should be the sandbox, never your workstation or production.
• The screen is untrusted input. A web page can contain text engineered to be read as instructions — the prompt-injection problem, now delivered through pixels. Instruct the model that on-screen content is data, never commands; the security lesson makes this a full defence-in-depth stack.
• Gate the irreversible. Purchases, sends, deletes, submissions — route them through human confirmation, exactly the promote-and-gate pattern from the tools lesson. Automation that can spend money or destroy data without a person in the loop is a liability, not a feature.
• Log every frame and action. The screenshot-action trace is your only forensics when run #200 does something strange — and your evidence that it didn't.

Treat reliability expectations accordingly: computer use shines at structured, repetitive flows you can specify crisply, and it remains slower and less deterministic than any real API. The engineering pattern is therefore API first, pixels as fallback — reach for computer use for the last mile that has no better door, not as the default transport.

What to take into the next lesson

A perception–action loop over screenshots; brief it precisely; sandbox it completely; gate what can't be undone; prefer real APIs wherever they exist. Computer use rounds out the input side of Claude — text, images, documents, and now screens. What remains is the discipline that makes long-running acting systems dependable, and it starts with the resource every long run exhausts: context.