Operate apps (computer use)

Preview

This page explains how Cowork’s 「PC の操作」 (computer use) reads a chosen app window and clicks or types in it. It covers setup, the limits on actions, how you take over, and how far it has been verified.

Under verification. Do not use it with business data

Computer use is at the stage where the OS integration is implemented and being verified.

• macOS: Screen capture and focusing a document have been verified on a real machine. End-to-end tests of clicking, Japanese text input, and saving have not passed yet.
• Windows: The capture and input components are implemented and have only been type-checked. Screen capture and input have not been verified on a real Windows machine.
• Actions chosen by a real model: Whether a real model can operate apps and finish tasks from work on this PC has not been verified. Platform-side work also remains before organizations can use image-capable models.

When you try it, use a new document that contains no confidential information.

How it works

1. You choose the app window to use in 「機能と接続 → PC の操作」 (Features and connections → Computer use).
2. In work on this PC, when the harness uses “capture the chosen app’s screen”, Cowork captures an image of that window only and passes it to the model.
3. The harness looks at the image and uses “operate the chosen app” to click, type, press keys, or scroll.
4. Before sending input to the OS, Cowork checks that the coordinates belong to the most recent capture and that the target window has not changed. After each action, it captures the screen again.

Cowork uses these OS facilities:

OS	Screen capture	Input
macOS 14 or later	ScreenCaptureKit	Accessibility and CGEvent
Windows 11	Windows Graphics Capture	UI Automation and SendInput

Setting up

1. Ask your administrator to confirm that 「PC の操作」 (computer use) is set to allow or confirm in the organization policy.

2. Open 「機能と接続 → PC の操作」. For 「画面を確認する」 (View the screen) and 「クリック・文字入力」 (Click and type), press 「OS の許可を設定」 (Set OS permission) and grant the permissions. On macOS, allow Hane Cowork under System Settings → Privacy & Security → Screen & System Audio Recording and Accessibility.

3. Under 「確認するアプリのウィンドウ」 (App window to check), choose the target from 「ウィンドウを選ぶ」 (Choose a window) and press 「選んだ画面を確認する」 (Check the chosen screen). Make sure only the chosen window appears. This check does not send the screen to the model.

4. Press 「この PC での作業に使う」 (Use for work on this PC). Work you start after this uses the window for screen capture and input.

The chosen window is not kept across app restarts. Choose it again after restarting.

Limits on actions

• Only the chosen window is operated. Input is refused if the target app or window changes.
• Typing into password fields is refused.
• Input is tied to the most recent capture. Input using coordinates from an older capture is refused.
• Input is refused while you have taken over and after the work is stopped.
• Input goes to the foreground window. It is not yet guaranteed that you can safely use the same PC in parallel. Detecting actions you take in other windows is not implemented.
• Cowork does not bypass UAC prompts or apps running with administrator rights.

Taking over and stopping

For work that captured the screen, a 「ワークスペース」 (Workspace) panel opens on the right of the work screen, and its 「PC の操作」 tab shows the most recent capture.

Button	Behavior
「自分で操作する」 (Take over)	Stops the agent’s input so you can operate yourself
「Hane に戻す」 (Hand back to Hane)	Ends your turn and hands control back to the agent
「止める」 (Stop)	Stops the work

Captured screen images are not saved to history.

Next steps

Confirmations and organization policyAllow, confirm, or deny computer use

Availability and limitationsVerification status of each feature