Benchmarks

Nine jobs only a phone can do

The result of each job cannot be produced by adb, am, cmd, settings, content or a web search. A script checks each one from the phone's own data, never from the agent's report. All tools ran on the same emulator on 10 November 2026 with Claude Sonnet 5.5 through claude -p, one agent at a time: phonedriver three passes, the others two. The phonedriver build is cfa37b2; release 0.1.0 is that build plus Windows build and test fixes. Totals are sums of per-job medians.

adb ran only the two jobs where any route is allowed: 4 of 4 runs with 203,751 tokens. phonedriver's sum for the same two jobs is 92,737.

Each job

Bars are median tokens, drawn to scale within each job. After the bar: median tokens, runs passed, median tool calls and median wall time.

Where phonedriver does not win

• phonedriver had the lowest median wall time on 8 of the nine jobs. Not on these: mobile-mcp on log-code (22 s against 31 s). Wall time includes the model's own thinking, so it moves with the day.
• These are two or three runs per cell on one emulator. They describe what happened and are not rates. Against its own 7 November runs phonedriver used 657,542 tokens where it had used 977,780, but the emulator was not the same one, so part of that is the device.

What each tool failed at, the versions, and every run that was thrown away and why are in docs/bench/jobs.md. The method is in Benchmarking AI agents on jobs only a phone can do, and the comparison write-up in phonedriver vs mobile-mcp vs phone-harness vs adb on real phone jobs. If you maintain one of these tools and think I set it up wrong, open an issue and I will rerun it.

Speed against adb and phone-harness

p50 of 50 runs after 20 warm-ups, on an Android 17 emulator (API 37, headless, software rendering). Everything ran on the emulator's own host with the three tools interleaved. phonedriver 0.1.1, 11 November 2026. Host CPU load was 31% to 80% around the rows, from other work on the machine. Bars are drawn to scale within each operation; the tick is p95.

Every operation, fastest gain first

How many times faster phonedriver is, then each tool's median time with its slowest 1 in 20 (p95) under it.

The range under each ratio is its 95% confidence interval.

On a real phone

A smaller pass on a HONOR X8 (Android 13) over USB, on 9 November 2026. 30 samples per phonedriver row and 20 per adb row; the two tools never ran at once.

The screen read was 461 bytes against 25,334 bytes of XML. On the phone's main Settings page, uiautomator dump failed with "could not get idle state" after 11.5 s, because the page never goes idle, while phonedriver read it in 110 ms. Typing and taps were not measured on the phone.

How to read it

• The bare tap is the closest row. Host to reply, phonedriver takes 38.5 ms when taps come back to back (4.4 times faster than adb) and 28.7 ms when they are 1.5 s apart (2.4 times). The tap misses its own 15 ms budget (decision D14), and typing 100 characters misses its 150 ms budget (158.3 ms).
• adb returns raw XML and a tap that returns before the app reacts. phonedriver returns a pruned, numbered tree and waits for the app. Different outputs for the same job. For "tap, then what changed", the adb row is a tap, one uiautomator dump and a diff in the caller.
• adb input text and phone-harness cannot type Unicode. phonedriver typed 100 mixed-script characters (ASCII, accents, CJK, Arabic, emoji) in 225.0 ms.
• Fresh daemon to first screen read: phonedriver 1,620.5 ms, phone-harness 2,999.4 ms (new Python process), adb 2,488.0 ms (one dump; adb has no cold start, so this is for scale only).
• phone-harness's helpers ran in process, which spares it a Python start per call and favours it.
• No iOS device is in this run. On the iOS 26.5 Simulator a tap takes 357 ms against 80 ms for the Simulator's own tool, because XCTest takes about 240 ms to confirm the event (D15). On an iPhone 14 Plus and an iPad Pro (M5) it takes 534 and 454 ms.

The earlier 153x figure came from a quieter, single-tool run and is not current. The full method, machine state and the history of the number are in the repo.

An earlier token run: three small tasks

Before the nine jobs, a smaller run: Claude Sonnet 5.5 through claude -p, each tool the only tool, three tasks checked from device state (the alarm database, the testbed event log, the answer text). Tokens are input, output, cache creation and cache read. Device: a 2560x1600 tablet emulator.

On the sum of medians phonedriver used 34% fewer tokens than phone-harness (218,357 against 330,375). It is ahead on the longest task, level on the form and 8% behind on the Wikipedia task, where individual runs ranged from 26,147 to 64,522 tokens. Raw adb finished alarm-gym in 2 calls with a SET_ALARM intent (18,607 tokens), a shortcut the task allows, and failed the web form, a WebView with no accessibility nodes. mobile-mcp failed every run of this earlier setup for an environment reason (its helper could not reach a remote adb server), so it is not compared here; the nine jobs above gave it its own adb server.

The fixed cost per turn is also measured: the first turn of a one-word claude -p costs 5,606 tokens with phonedriver's 14 MCP tools and 5,214 with phone-harness. All three rounds, the fixes between them and the caveats.

Raw data and reproduction

• jobs.json: every agent run of the nine jobs, with its pass, tokens, tool calls and wall time.
• speed-samples.csv and second-pass samples: every timed sample.
• speed.json: p50, p95 and bootstrap intervals per operation. speed-meta.json: machine and versions.
• tokens.json: every run of the earlier three-task benchmark.
• Repeat the speed run: PH_PYTHON=<python with phone-harness 0.3.3> bash bench/speed/reproduce.sh in the repository.