Automation Best Practices

Tool names used below (key, type_text, screenshot, …) are the desktop tools defined in tools-reference.md. Anything described as a client-side pause or loop runs in your agent’s own code, not on the desktop.

Minimize screenshots (the #1 rule)

Screenshots are large image payloads. A naive agent screenshots after every click and burns tokens for no benefit.

• Take one screenshot to establish state, perform 3–5 actions, then screenshot only to verify a meaningful unit of work.
• Trust predictable UI actions. Tool/menu selections, color/dropdown settings, typing, and keyboard shortcuts reliably succeed — do not screenshot to confirm them. Proceed directly to the next action.
• Only screenshot after completing a batch, or when something unexpected happens and you need to diagnose.
• Treat a screenshot budget as a hard target: if you are screenshotting after single actions, you are doing it wrong.

Batch actions

Group related actions into one sequence with no intermediate screenshots:

• Setup then act: select tool → set options → act — no screenshot between setup steps.
• Repeat similar actions together: e.g. fill five form fields or draw four shapes in sequence, then one screenshot at the end — not one per element.

Plan coordinates up front

• From the establishing screenshot, note reference points (window corners, toolbar edges, canvas origin) and derive all target coordinates before acting.
• For drags, compute both start and end coordinates before beginning.
• Don’t discover coordinates by trial and error — it wastes screenshots.

Launch applications reliably

• Prefer the Run dialog: key("super+r") → type_text("notepad") → key("Return"). It is more reliable than Start-menu search.
• If the fleet exposes Application Catalog tools (launch_application), use those instead of navigating the shell.
• After launching, pause 3–5 seconds client-side before interacting so the app is ready (e.g. time.sleep(3) / await asyncio.sleep(3) in your agent loop — the desktop toolset has no wait tool, unlike some computer-use APIs).

Prefer forwarded tools for file/web work

If the fleet has MCP tool forwarding enabled, use forwarded filesystem/fetch tools instead of driving an app by pixels — reading a file with a forwarded tool is far more reliable than opening it and reading the screen. See tool-forwarding.md.

Handle unexpected dialogs

Apps show update prompts, recovery dialogs, setup wizards. When one appears:

• key("Escape") to dismiss, or key("alt+F4") to close a window.
• For save prompts choose “Don’t Save”/”Discard” unless the task requires saving.
• key("alt+Tab") to bring the target window back to focus.

Don’t repeat failures

If an action fails twice, change approach — take a screenshot to re-orient, try a different launch path or coordinates, or use a forwarded tool. Repeating the same failing action wastes budget.

Verify by outcome, not mechanics

To confirm a task succeeded, check the produced artifact (a saved file via a forwarded filesystem tool, an uploaded screenshot in S3, a changed window title) rather than screenshotting every intermediate step.