Skip to content

Computer Use

computer_use lets agents inspect and operate a host desktop when browser automation is the wrong shape: native apps, sensitive logged-in sites, and permission or login flows that need the real local desktop.

The agent container never receives raw desktop access. A backend-neutral router accepts the request through Pynchy's policy-enforced IPC boundary, preserves the workspace attribution, and sends it to an available provider plugin on the host. This makes computer use optional and replaceable on every platform.

Configure provider order

The built-in router tries providers in the configured order. Its defaults are Peekaboo followed by Cua Driver:

[plugins.computer-use.options]
providers = ["peekaboo", "cua-driver"]

Provider fallback happens only when a provider is unavailable. Pynchy does not retry a failed action through another provider because doing so could repeat a partially completed click, keystroke, or other mutation.

Both built-in providers require macOS. On Linux or another host platform, disable them and install a provider plugin for that platform; the computer_use tool and its policy contract do not change.

[plugins.peekaboo]
enabled = false

[plugins.cua-driver]
enabled = false

[plugins.computer-use.options]
providers = ["my-linux-provider"]

Built-in: Peekaboo

Peekaboo is the preferred macOS provider. It offers semantic accessibility snapshots, stable element references, application and window targeting, menus, dialogs, clipboard operations, and Spaces management.

Install it on the macOS host and inspect its permissions:

brew install steipete/tap/peekaboo
peekaboo permissions status

Grant Screen Recording and Accessibility in macOS System Settings. Coordinate clicks, keyboard input, and synthetic click fallback can also require Event Synthesizing permission.

Override its executable or timeout when necessary:

[plugins.peekaboo.options]
binary = "/opt/homebrew/bin/peekaboo"
timeout_seconds = 30

Start by listing applications, then target an application or PID when listing windows:

computer_use(action="list_apps")
computer_use(action="list_windows", app="TextEdit")
computer_use(action="capture", app="TextEdit", label="editor")

A capture returns a semantic snapshot ID, stable element references such as B1 or T2, and a PNG artifact. Reuse the snapshot ID and element reference for the next action:

computer_use(action="click", app="TextEdit", snapshot_id="SNAPSHOT", element="B1")
computer_use(action="set_value", snapshot_id="SNAPSHOT", element="T2", value="hello")

The provider also supports:

  • clicks, text, shortcuts, scrolling, accessibility actions, and value changes
  • application launch and application/window discovery
  • menu discovery and selection
  • dialog discovery, buttons, text input, file selection, and dismissal
  • clipboard get, set, clear, save, and restore
  • Space discovery, switching, and moving windows between Spaces

Built-in: Cua Driver

Cua Driver remains available as a compatibility provider. Install it on the macOS host and verify its daemon and permissions:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)"
cua-driver call check_permissions
cua-driver status

[plugins.cua-driver.options]
binary = "cua-driver"
timeout_seconds = 30

Cua Driver supports captures, application/window discovery, application launch, numeric-element or coordinate clicks, text, shortcuts, scrolling, and permission checks. Stable element references, menus, dialogs, clipboard, and Spaces require a richer provider such as Peekaboo.

Compose unrestricted computer use into profiles

Define one reusable profile when multiple workspaces should operate the host desktop without runtime approval:

[tools.computer_use]
type = "builtin"
name = "computer_use"

[profiles.unrestricted-computer-use]
tools = ["computer_use"]

[profiles.unrestricted-computer-use.capabilities."desktop.computer.use"]
decision = "allow"

[profiles.desktop-worker]
includes = ["base", "unrestricted-computer-use"]

Register the builtin tool once, then reuse the profile wherever needed. The tool selection makes computer_use available. The explicit capability rule authorizes it without the normal session prompt. The built-in computer-use plugin supplies its core instructional skill automatically, so the profile does not need a separate skills entry.

Compose this profile only into workspaces intended to control the real host without confirmation. A service property set to "forbidden" still blocks the action. See Capability Rules for the full policy precedence.

Artifacts and safety

Screenshots are saved under the workspace IPC directory and exposed inside the agent container at /workspace/ipc/computer-use/<file>.png.

This tool controls the real host desktop. Gate it with workspace security policy for non-admin workspaces. When policy requires approval, the approval covers only computer_use for the active agent session and clears when that session ends. Other tools keep their own approval scope, and no approval overrides a later policy denial. Agents should not enter secrets, payment details, 2FA codes, or destructive confirmations unless the user explicitly authorized that exact action.

Want to customize this? Write your own provider plugin using the pynchy_computer_use_backend hook, or open a feature request.