How to replay an agent's run in CI with GitHub Actions

Save what an agent did on your app as a steps file, replay it with no model, and run the files on every commit with the phonedriver GitHub Action on an Android emulator or an iOS Simulator.

An agent that finds its way through your app once has written a test, if you keep what it did. phonedriver save writes the session's steps to a file, phonedriver replay runs them again with no model, and the phonedriver GitHub Action replays the files on an emulator or a Simulator on every commit.

Save a run

After the agent (or you, with the CLI) has walked a flow that works:

phonedriver save checkout.steps

save lists every step it wrote, one a line, so a step that was a wrong turn can be deleted from the file. Only calls that worked are kept: taps, typing, waits, launches, shell commands, settings and permissions. An element is kept by its text and role, not by its @ref, so the file works in a later session and on another device with the same app. Save once before the flow to a file you throw away, so the second file holds only the flow.

A wait step is what makes a replay check something: phonedriver wait "Order placed" after the last tap fails the replay if the text never shows.

Replay it

$ phonedriver replay checkout.steps
replayed 9 steps in 12.4 s

A replay stops at the first step that fails, or at a step that changed the screen when it was saved and now changes nothing. It prints the steps that ran, the screen as it is, and where to go on from:

error: step 6 of 9 (tap "Pay now") failed: no element matches "Pay now"
hint: Do step 6 yourself, then replay the same file from step 7 for the 3 steps left.

Text typed with --secret is never written to the file, so a replay stops at that step. Keep real passwords out of runs meant for CI and type test values in plain text.

Check what each screen shows

A replay checks that the taps land and the waited-for text shows. --screens also checks what each step left on the screen:

$ phonedriver replay settings.steps --screens screens/settings
replayed 5 steps in 10.5 s
kept 5 screens in screens/settings; a replay with --screens screens/settings compares each step's screen with them

The first replay keeps a screenshot after each step. Later replays compare each step's screen with its screenshot and stop at the first step whose text changed, with the text that changed:

error: step 5 of 5 (wait "Name") failed: its screen is not the one in screens/settings/05.png
changed: 2 regions, 35.7% of the screen
~ "General" → "About" at 201,85
+ "Name" at 59,161
hint: If the new screen is right, delete screens/settings/05.png and replay again to keep it.

Pixels that change with no text in them, such as a pressed row or a scroll bar, do not stop a replay, and the status bar is left out: its clock, notification icons and battery change on their own. A screen that differs is looked at again until it stops moving, so an animation that had not finished does not fail the job. Keep the screens on the device type CI runs, since a screenshot of another size stops the replay.

Run the files in GitHub Actions

The phonedriver repository is also an Action. It starts an Android emulator on a Linux runner or an iOS Simulator on a macOS runner, installs your build, replays each file and fails the job at the first run that stops:

name: replay
on: [push, pull_request]
jobs:
  android:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./gradlew assembleDebug
      - uses: hridaya423/phonedriver@v0.1.5
        with:
          app: app/build/outputs/apk/debug/app-debug.apk
          runs: e2e/*.steps
          record: true

For iOS, use a macOS runner, platform: ios and a Simulator .app built with xcodebuild -sdk iphonesimulator. device picks the Simulator; the default is the first iPhone the runner has.

Each run's result is a row in the job summary. The reply of every run, and the screen and app logs of a run that stopped, go in an artifact; with record: true the artifact has a video of every run too.

InputDefaultWhat it does
runsSteps files, one path or glob a line
platformandroidandroid (Linux runner) or ios (macOS runner)
appAn .apk or a Simulator .app to install first
api-level35The Android system image
devicefirst iPhoneThe Simulator to boot
recordfalsetrue keeps a video of every run
screensA folder of kept screens, a subfolder for each run named as its steps file. A run with no subfolder keeps its screens in the artifact, under screens/, for you to commit
versionlatestThe phonedriver release to install
artifactphonedriver-<job>The artifact's name

The outputs passed and failed count the runs.

Limits

  • Text that changes on its own, such as free space, a date or the keyboard's suggestions, stops a replay with --screens. Use it on flows whose screens stay the same from one replay to the next.
  • A replay finds each element again by its text. A run saved on one language or screen size can miss on another; save the run on the device size and language CI uses.
  • The Action runs the Android emulator only on Linux runners, which have hardware acceleration; GitHub's macOS runners cannot run it.
  • A step that waited on something outside the app (an email, a push notification) has nothing to wait on in CI.