End-to-end testing of mobile apps

Google released a new version of the Google Auth library.

It had a bug that broke Google Login on Android for API 26 and earlier.

This impacted my small but popular Music Player app in Google Play.

It reveals the kind of problem, I have alluded to in end-to-end testing.

The only way to avoid such problems is by end-to-end testing.

However, I find all Android testing frameworks to be terrible.

Most of the frameworks are focused on testing the underlying code.

However, I’m interested in UI interactions to prove the user can accomplish the job.

And that’s how I came across mobile.dev.

Mobile.dev has an excellent open-source framework called Maestro which one can use to write intuitive tests.

You describe how the UI interactions and expectations.

And the framework just verifies that.

Here’s a sample for the test I wrote to prevent my app from breaking in the future.

This snippet describes a basic test that I wrote to assert that Google Drive Folder sharing is always working.

Yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Same this file inside ".maestro" folder in the repo root
---
# Ref: https://maestro.mobile.dev/api-reference
appId: "net.ashishb.androidmusicplayer"
---
- launchApp
- waitForAnimationToEnd

- tapOn:
    id: "shareable_link"
- inputText: "https://drive.google.com/drive/folders/<folder-id>"
- tapOn: "Link Google Drive Folder"
- waitForAnimationToEnd

- assertVisible: "Cached Files"
- assertVisible: "Recently Added"
- assertVisible: "Local Files"
- assertVisible: "song.mp3"
- assertVisible: "Files: 1,0*.*"
- assertVisible: "Files: 1,55.*"

# Now download a file and play it
- tapOn: "song.mp3"
- waitForAnimationToEnd

- assertNotVisible:
    id: "rewind_button"
# Verify bottom sheet expands
- tapOn:
    id: "music_status"
- assertVisible:
    id: "artist_name"
...

And here’s a simple GitHub Actions code to run all the tests on mobile.dev

1
2
3
4
5
6
7
# Run all test flows in ".maestro" directory
- name: Run end to end tests with mobile.dev
  uses: mobile-dev-inc/action-maestro-cloud@v1
  with:
  # The API key can be found at https://console.mobile.dev/quickstart
          api-key: ${{ secrets.MAESTRO_CLOUD_API_KEY }}
          app-file: <path-to-apk>

Note:

I would love to run these tests directly on GitHub Actions but Maestro consumes too much memory and won’t run on `macos-latest`. Maybe it will run on macos-latest-xl but I never tested it.
You can run maestro CLI locally against a connected device/emulator.
For the Google Auth bug, I still cannot test using Maestro as it only supports recent Android versions. However, the Maestro testing prevented me from shipping another Gradle-related bug.

Related Posts