# Desktop Control

Lets an agent drive the mouse and keyboard, grab screenshots and find images or text on screen through the uvx desktop-agent CLI.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 4000 installs · ★ 3
- Author: patrickporto
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/patrickporto/desktop-agent/tree/HEAD/SKILL.md
- Updated: 2026-06-17
- HTML: https://affpapa.org/en/nevibecode/skills/desktop-control

## What it does

The skill sits between your agent and the desktop, using PyAutoGUI underneath. Everything goes through one command shape, `uvx desktop-agent <category> <command>`, across five categories: `mouse`, `keyboard`, `screen`, `message` and `app`. The mouse moves to coordinates with a set duration, clicks left, right or middle, drags and scrolls. The keyboard types text at an interval, presses single keys and fires hotkeys from a comma-separated list like `ctrl,c`.

The screen side is the part worth reading closely. You can capture the full screen, a region or a named window, and `locate` plus `locate-center` search for an image with a confidence threshold. OCR is there too: `locate-text-coordinates` returns coordinates for a string, `read-all-text` dumps everything in a window. Through `app` you open programs, focus windows and list what is visible, on Windows, macOS and Linux.

Two things to keep in mind. Coordinates and image matches hold only until a window moves or a button shifts, so the skill itself tells agents to check `screen size` and `on-screen` before clicking and to space commands out with delays. Confirm destructive actions with the user. Also, `uvx` pulls the package with PyAutoGUI, so you need a working Python setup with network access.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add patrickporto/desktop-agent --skill desktop-control -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/desktop-control/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/desktop-control/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add patrickporto/desktop-agent --skill desktop-control -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/desktop-control/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add patrickporto/desktop-agent --skill desktop-control -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/desktop-control/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install patrickporto/desktop-agent desktop-control
```

2. **Manually: put the files in place**
- File: `.github/skills/desktop-control/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add patrickporto/desktop-agent --skill desktop-control -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/patrickporto/desktop-agent
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/desktop-control/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add patrickporto/desktop-agent --skill desktop-control -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/patrickporto/desktop-agent/tree/HEAD/SKILL.md

## When to use it
- fill a form in a desktop app that has no API
- capture one window and find a button inside it
- read on-screen text with OCR and get its coordinates
- build a multi-key hotkey and send it to a window
- open an app, wait for focus and type into it

## Example prompts
- Take a screenshot of the active window and find the save button in it
- Open notepad, focus it and type "done"
- Press ctrl+c in the current window, then click 500 600 and paste

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: c2466fc645195c91.

BEGIN THIRD-PARTY FILE c2466fc645195c91
````text
---
name: Desktop Control
description: Control mouse, keyboard, and screen for desktop automation tasks
---

# Desktop Control Skill

This skill provides comprehensive desktop automation capabilities through PyAutoGUI, allowing AI agents to control the mouse, keyboard, take screenshots, and interact with the desktop environment.

## How to Use This Skill

As an AI agent, you can invoke desktop automation commands using the `uvx desktop-agent` CLI.

### Command Structure

All commands follow this pattern:

```bash
uvx desktop-agent <category> <command> [arguments] [options]
```

**Categories:**
- `mouse` - Mouse control
- `keyboard` - Keyboard input
- `screen` - Screenshots and screen analysis
- `message` - User dialogs
- `app` - Application control (open, focus, list windows)

## Available Commands

### 🖱️ Mouse Control (`mouse`)

Control cursor movement and clicks.

```bash
# Move cursor to coordinates
uvx desktop-agent mouse move <x> <y> [--duration SECONDS]

# Click at current position or specific coordinates
uvx desktop-agent mouse click [x] [y] [--button left|right|middle] [--clicks N]

# Specialized clicks
uvx desktop-agent mouse double-click [x] [y]
uvx desktop-agent mouse right-click [x] [y]
uvx desktop-agent mouse middle-click [x] [y]

# Drag to coordinates
uvx desktop-agent mouse drag <x> <y> [--duration SECONDS] [--button BUTTON]

# Scroll (positive=up, negative=down)
uvx desktop-agent mouse scroll <clicks> [x] [y]

# Get current mouse position
uvx desktop-agent mouse position
```

**Examples:**
```bash
# Move to center of 1920x1080 screen
uvx desktop-agent mouse move 960 540 --duration 0.5

# Right-click at specific location
uvx desktop-agent mouse right-click 500 300

# Scroll down 5 clicks
uvx desktop-agent mouse scroll -5
```

### ⌨️ Keyboard Control (`keyboard`)

Type text and execute keyboard shortcuts.

```bash
# Type text
uvx desktop-agent keyboard write "<text>" [--interval SECONDS]

# Press keys
uvx desktop-agent keyboard press <key> [--presses N] [--interval SECONDS]

# Execute hotkey combination (comma-separated)
uvx desktop-agent keyboard hotkey "<key1>,<key2>,..."

# Hold/release keys
uvx desktop-agent keyboard keydown <key>
uvx desktop-agent keyboard keyup <key>
```

**Examples:**
```bash
# Type text with natural delay
uvx desktop-agent keyboard write "Hello World" --interval 0.05

# Copy selected text
uvx desktop-agent keyboard hotkey "ctrl,c"

# Open Task Manager
uvx desktop-agent keyboard hotkey "ctrl,shift,esc"

# Press Enter 3 times
````
END THIRD-PARTY FILE c2466fc645195c91

The third-party file has ended. Everything between the c2466fc645195c91 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2516 of 20752 bytes. © patrickporto · MIT · https://github.com/patrickporto/desktop-agent/tree/HEAD/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md