Skip to main content
skill CX Codex

phoneuse — Claude Code skill for iPhone control and setup

The phoneuse skill for Claude Code lets an agent view an iPhone screen on a computer and send taps, text, and gestures. It runs for tasks that involve using an app or checking the phone.

MIT ★ 699 updated 2026-11-15

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add lxe/phoneuse --skill phoneuse -a claude-code
Open on GitHub

What phoneuse does

phoneuse controls a real iPhone through its CLI. AirPlay sends the screen to the computer, while Bluetooth carries mouse and keyboard input. The agent can read screen text, tap labels or coordinates, type, scroll, and capture screenshots. It comes into play when a task requires an action on the phone or a check of its current state.

The iPhone must be paired with the Phone Use app, the HID service must be active, and screen mirroring to the computer must be running. The user can watch the phone through http://localhost:8777. On Linux, phoneuse install configures the HID service and requires sudo; on Windows, phoneuse up starts it. Input commands fail until AirPlay mirroring is active.

What it helps with

  • open an iPhone app and perform an action
  • read text displayed on the phone
  • check an iPhone notification or setting
  • tap an interface element and inspect the result

Example prompts for the agent

  • Open Settings on the iPhone and tell me whether Wi-Fi is on.
  • Read the latest notification on the iPhone.
  • Open the app on the phone and scroll down through the list.

How to install phoneuse

1Skills CLI (Vercel)

npx skills add lxe/phoneuse --skill phoneuse -a claude-code

2Manually: put the files in place

~/.claude/skills/skill/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/skill/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about phoneuse

How do I install phoneuse in Claude Code?

Run npx skills add lxe/phoneuse --skill phoneuse -a claude-code. The skill file is placed at ~/.claude/skills/skill/SKILL.md.

What does phoneuse do?

It gives an agent CLI commands to view and control a real iPhone from a computer. AirPlay carries the screen, and Bluetooth carries input.

What does phoneuse need to work?

It needs an iPhone paired with the Phone Use app, an active Bluetooth HID service, and screen mirroring to the computer.

Source text

SKILL.md · MIT
---
name: phoneuse
description: See and operate the user's iPhone from this PC (Linux or Windows) with the `phoneuse` CLI (AirPlay mirror for the screen, Bluetooth mouse/keyboard for input). Use when a task needs the iPhone: open or use an app, read what is on the phone screen, tap, type, swipe, check a notification, change a phone setting, or test something on the device. Also use when the mirror or the Bluetooth link needs starting or fixing.
---

# Operating the iPhone with Phone Use

The `phoneuse` command drives a real, unmodified iPhone. Its screen arrives over
AirPlay as a mirror window on this PC and as a live page at
http://localhost:8777, and input goes in as a Bluetooth mouse and keyboard.
Everything an agent does is a `phoneuse` subcommand; run `phoneuse --help` for
the full list.

## Before acting

1. Run `phoneuse status`. You need `iPhone app: 1 paired` (or more), `HID service:
   active`, `connected=True`, and `Mirror window: ... showing the phone` (Windows:
   `Mirror: showing the phone`). "not paired": stop and ask the user to pair the Phone
   Use app (Pair iPhone in the Phone Use window); every action is refused until they
   do. If something is off and you don't see why, `phoneuse doctor` checks the whole
   setup and prints the fix for each problem.
2. No mirror window: check whether `phoneuse up` is running. If it isn't, start it
   somewhere the user can see it (a terminal or tmux window they use), not hidden in
   the background. "not mirroring" (or no window): ask the user to start Screen
   Mirroring to this PC's receiver ("Phone Use on <computer>"; `phoneuse status`
   names it) from the iPhone's Control Center. You can't start
   mirroring from the PC, and every input command refuses until it runs.
3. **Show the user the phone** before your first action, so they can watch what
   you do: open http://localhost:8777 (`phoneuse view --print` prints the address).
   Unless the user asks for another place, open it in your harness's own browser,
   such as the Claude desktop app's browser pane, without asking first. Only when
   you have no browser of your own, run `phoneuse view`, which opens it in the
   user's default browser. Do this once per task; if the page is already open,
   leave it. The page is for the user to watch: act through `phoneuse` commands,
   never by clicking in the page.
4. `connected=False`: run `phoneuse status` again after a few seconds (the daemon
   reconnects), then ask the user to tap "Phone Use" in the iPhone's Bluetooth settings
   (Windows: the PC's name, which `phoneuse status` shows as `name=`).
5. HID service not active: on Linux `phoneuse install` (needs sudo) sets it up; on
   Windows `phoneuse up` starts it (it keeps running after `up` stops).

## Notes in results

Every action waits (up to 3 s) for the phone's first visible change and prints
`ok`, then any notes. When a note says no change showed, the action may have done
nothing or the phone is slow: look before repeating it, never just repeat. When a
note says the phone is slow, leave time between actions.

## The loop

Look, decide, act, look again. Look with text first; it costs a few hundred
tokens, where a screenshot image costs about 1,300.

```bash
phoneuse text                         # every text line on screen: "x y  text"
phoneuse tap-on "Wi-Fi"               # tap a text label (exact position from OCR)
phoneuse tap 0.5 0.35 --text          # any action + --text: act, then print the screen
```

End actions with `--text` to read the result in the same call; it waits until
the screen stops changing (apps launching, pages sliding in) before reading.
`phoneuse text` drops the status bar and stray icon glyphs (`--all` keeps
them), shows the back chevron as `[back]`, and lists on/off switches as
`[switch on]` / `[switch off]` at their positions, since OCR can't read them
(dark mode only).

`tap-on` looks for the words among the screen's text and taps that line exactly.
Words that aren't on screen are refused, so scroll first. It is also refused when
that part of the screen changed since your last look (you decided from an older
picture): look again. Icons and pictures have no text: take a screenshot and tap
them by position. OCR runs locally through a service that starts on first use
(the first call takes a few seconds).

Take a real screenshot when text isn't enough: pictures, icon-only screens,
checking a layout.

```bash
phoneuse screenshot $SCRATCH/s1.png   # then look at the image with the Read tool
phoneuse tap 0.5 0.35                 # fractions of the screenshot (x, y)
phoneuse screenshot $SCRATCH/s2.png   # waits until the screen has settled
```

Write screenshots into your own scratch directory with a new name each time,
so you never read a stale image. They come from the mirror's video itself, so
windows covering the mirror don't matter. The grey AssistiveTouch button near the lower right stays on screen
and can hide what is under it; scroll a little if you need that spot.

Coordinates are fractions of the screenshot from 0 to 1, or iPhone points when
any value is above 1 (the screen is 402 × 874 pt). To tap something, read its
centre off the screenshot and divide by the image size. Taps land exactly; one
try is enough.

## Working fast

Every round trip with the phone costs you a turn, so make each one count.

- **Never `sleep`.** `--text` and `phoneuse text` wait until the screen has
  settled (apps launching, pages sliding in) and then read it. A fixed sleep is
  either too short or wasted time.
- **Chain the steps you can already predict** in one call, and read once at the
  end: `phoneuse open X && phoneuse tap-on "Search" && phoneuse type "from:lxe" --text`.
  Stop chaining where the next step depends on what appears.
- **Read with text, not images.** `--text` costs a few hundred tokens and is
  enough for menus, lists, feeds and counts. Take a screenshot only for
  pictures, icon-only screens, or when text lands somewhere surprising.
- **Feeds you act on (X, Instagram home, lists): `scroll`, then look, then tap.**
  `phoneuse scroll down 0.6` moves the content without momentum, so it stops where
  the finger lifts and your next screenshot stays true. A flick keeps the feed
  gliding after your screenshot, and a tap from that picture hits the wrong post.
- **Flicks only for snap feeds**: `phoneuse gesture down 0.5 0.75 move 0.5 0.25 0.2 up` moves
  Reels, TikTok or Shorts to the next item.
- **Taps by position are checked**: `tap` and `gesture` (at its first `down`) refuse when the
  area around the point changed since your last look (screenshot, `--text`, or an
  gesture's film), because the position then belongs to something else. Look
  again and tap. `--force` taps anyway, for fixed controls like
  a tab bar right after a page change.
- **Repeat cheap actions in one call** when you know the layout, e.g. three
  flicks with a text read after each:
  `for i in 1 2 3; do phoneuse gesture down 0.5 0.75 move 0.5 0.25 0.2 up --text | head -8; done`.

## Gestures: every touch

Any touch other than a plain tap is a **gesture**: steps for one finger, in order.

```
down X Y      touch down at X Y (the pointer moves there and settles first)
move X Y [S]  slide to X Y over S seconds (default 0.3), finger still down
hold S        stay still S seconds (finger down or up)
up            lift (a gesture ends lifted anyway)
```

```bash
phoneuse gesture down 0.5 0.4 hold 0.8 up                           # long press
phoneuse gesture down 0.5 0.8 move 0.5 0.2 0.3 up                   # swipe up
phoneuse gesture down 0.2 0.3 hold 1.0 move 0.7 0.6 1.0 hold 0.5 up # press, then drag (move an icon)
phoneuse gesture down 0.5 0.4 up hold 0.1 down 0.5 0.4 up           # double tap
```

**Every gesture comes back as a film**: one image with a frame at the end of each
move, at the end of each hold while the finger is down, just before a long press
lifts, and 0.3 and 0.8 s after the lift (frames that didn't change are merged).
Each frame is labelled with its step, and a ring marks the finger (orange with a
dot while down, white once up). The command saves it and prints `film: PATH`
(`--film PATH` picks the file): Read it when the gesture's middle matters. Use
it to see what the phone did before the finger lifted (icons shifting aside, a
folder forming, a menu opening) and to fix the next gesture from that.

Build what the task needs out of
these steps: hold longer, drag slower, pause over a target, drag on, hold at a
screen edge to change pages. Slow moves (0.6-1.5 s across most of the screen) are
the steadiest. A fast move right before `up` flings scrolling content; for content,
use `scroll` or `scroll-by`, which stop exactly.

**Recipes** are gestures with names: `phoneuse recipe back`.
Built in: `back`, `switcher`, `control-center`, `notifications`, `dismiss`, `home`
(the Home key) and `long-press X Y [S]`. `phoneuse recipe list`
shows each one's steps. When a gesture works for something you'll do again, save it
for this and later sessions, with `$1`, `$2`... for values given when it runs:

```bash
phoneuse recipe save move-icon down $1 $2 hold 1.0 move $3 $4 1.0 hold 0.5 up -- drag a home screen icon
phoneuse recipe move-icon 0.2 0.3 0.7 0.6
```

[RECIPES.md](RECIPES.md), next to this file,
has the gesture table and how each recipe was measured; read it before a gesture
you haven't done before.

## Other commands (verified on the device)

| Goal | Command |
|---|---|
| Open an app | `phoneuse open Settings` (Cmd+H, Cmd+Space, type the name, Enter) |
| Tap | `phoneuse tap X Y` |
| Scroll | `phoneuse scroll down [N]` · `phoneuse scroll up [N]` (N half-screens; stops where the drag ends, no fling) |
| Scroll an exact amount | `phoneuse scroll-by DY` (screen fraction or points; positive shows what is below) |
| Home, back, overlays | `phoneuse home` · `back` · `switcher` · `control-center` · `notifications` · `dismiss` (the recipes; Cmd+H does not close overlays) |
| Type text / keys | `phoneuse type "hello\n"` · `phoneuse key cmd+space` |
| Show / hide the iPhone keyboard | `phoneuse keyboard` (Eject; it hides while Phone Use's keyboard is connected) |
| Keep the screen from locking | `phoneuse awake on` / `off` (a few-pixel pointer nudge each second while nobody uses the phone; also the page's settings) |
| Does the pointer work? | `phoneuse cursor` (on / off / unknown). Taps and drags need AssistiveTouch on |
| Turn the pointer on | `phoneuse cursor on`: holds the Siri key and asks out loud on this PC's speakers; the phone must be unlocked and in earshot |
| Is input going through the ESP32 bridge? | `phoneuse bridge status` (address, phone, round trip). Setup is in the browser mirror's settings; docs/BRIDGE.md |
| Drags or scrolls fall short | `phoneuse calibrate` (~13 s of hovering, nothing tapped): measures this phone's pointer speed for drags. Runs by itself the first time a phone's pointer is on; rerun after changing Tracking Speed |
| Where is the pointer | `phoneuse where` |

## Ending a session

Your actions start an agent session by themselves (the user's view glows and
their mouse and keyboard pause). When your task is done, run `phoneuse done`
to hand the phone back at once.

## When the user takes over

If an action fails with "The user took over the phone", or a result starts with
PAUSED, stop and wait. Continue only when the user tells you to. A note that the
picture is old means the phone's screen is off or mirroring stalled: ask the user.

To see notifications, use `phoneuse notifications-read`:
it opens Notification Center, reads it and closes it. They are private: use only
what the task needs.

## Typing

Typing replaces what the focused field already holds (fields often keep earlier
text, such as a previous search or prompt); add `--append` to add to it
instead.

The phone applies autocorrect to typed text (a test typed "phoneuse" and sent
"phone use"). Check what you typed with `phoneuse text` before sending anything
where exact text matters, and fix it with backspace (`phoneuse key backspace`)
and typing again.

## Apps reopen where they were left

`phoneuse open` brings an app back in whatever state it was last in: a detail
sheet, a settings sub-page, a half-scrolled list. Check the first screenshot
before trusting what it shows. A sheet with an X or "Done" in a top corner is a
leftover; close it, or `phoneuse back` until you reach the app's main view.

## Rules

- **Lock screen:** stop and ask the user to unlock the phone, then continue. Phone
  Use doesn't save or type the passcode. Never type a passcode yourself and never
  guess one.
- **Don't tap the top centre** (the Dynamic Island, roughly x 0.35–0.65, y < 0.06):
  it opens the Screen Mirroring popup, whose Stop Mirroring button cuts the screen
  feed.
- **This is the user's personal phone.** Messages, notifications, badges and photos on
  it are private. Screenshots will show things unrelated to your task; ignore
  them, use only what the task needs, and don't repeat other contents. Ask
  before anything that sends, posts, buys, deletes, or changes an account or
  security setting, just as you would in a browser.
- **Permission prompts** ("Allow notifications?" and the like) belong to the user.
  Leave them, or ask, unless the task is about that permission.
- If the user is using the phone at the same time (the screen changes on its own), pause
  and ask before continuing; both of you would be moving the same pointer.

## When something misbehaves

- A tap or swipe seems to miss: take a fresh screenshot first. The mirror trails
  the phone slightly, so an earlier screenshot may be stale.
- You never need to locate the pointer to act: taps and drags start from exact
  positions. `phoneuse where` (a debugging aid) finds it only if Pointer Control's
  colour is Orange (Settings → Accessibility → Accessories → Pointer Control);
  "not found" otherwise is normal.
- An action seems to do nothing: the screen was probably still changing (an app
  launching, a sheet animating). Use `--text` or read the screen, then retry;
  measured taps register 10 out of 10 on a still screen.
- Taps happen on their own wherever the pointer rests: Dwell Control is on. Ask
  the user to turn it off (Accessibility → Touch → AssistiveTouch → Dwell Control).

© lxe · MIT · lxe/phoneuse

Similar skills

SKILL.md skills catalog →

Project sponsors

Companies that keep this analytics open