I’ve spent the last six months or so testing mobile apps for both iOS and Android. Here’s eight of my key lessons learned:
- Automated UI testing tools for mobile apps are immature: whilst tools like WebDriver for automated UI testing of web apps are very mature, automated UI testing of native mobile apps is the opposite. Whilst Appium shows some promise as a cross-platform UI testing tool for mobile apps, in my opinion, the maintenance overhead is still high as the tests are quite flaky. I recommend more thorough automated unit and integration testing, possibly supplemented with WebView tests (see below) and some exploratory manual app testing of the UI.
- Base your mobile OS testing on projected usage, not current usage. Mobile hardware and OS upgrade cycles are shorter than desktop hardware, and if you look at trends and test against those, chances are you will have more realistic usage based testing.
This chart from mixpanel shows iOS7 adoption across a very large number of devices. You can see iOS6 is trending downwards as iOS7 is broadly adopted, so you could put less focus on testing iOS6.
- Test on real devices: I prefer to test on real devices like phones and tablets so that I can get a realistic feel for how the app runs. There’s also some things that can’t be tested on emulators/simulators like push notifications which require a device id, and Apple VoiceOver for accessibility testing which isn’t available in the simulators.
I have found that iPod Touch’s make excellent test iOS devices as they’re a lot cheaper than iPhones, and you can buy them refurbished on the Apple store, although if you want to test in the wild (see below), you’ll need a cellular iOS device. If you’re after an iOS6 specific test device you’re only option is to buy a 4th generation iPod touch refurbished as all new Apple iOS devices now come with iOS7 pre-installed.
It’s easy to find cheap Android pre-paid phones for testing, and you can still buy Android 2.3 phones with lower-res screens which is good for older version compatibility testing.
- Enable wireless distribution/installation of your test application locally: to enable quick installation of updates to your app on both iOS and Android.
For wireless iOS distribution this involves have a build machine (which has to be a Mac) compile an IPA file which is copied to a web-server, along with an XML manifest (PLIST) file that describes the app. You then have a simple web page with a itms-services link to the manifest, which can be opened in Safari on iOS, prompting the user to automatically install. If you sign your iOS app using an enterprise account it will run on an unlimited number of iOS devices without registering them as test devices. Full details are here.
For wireless Android distribution it is much easier. Simply have your build server compile an APK file and copy it to a web-server along with a page linking to the APK directly. Any Android phone set to allow installation of apps outside the Play Store will download the file and prompt to install it. Voila.
- Test in the wild: I like testing during my bus/train ride home to see how my app handles with intermittent cellular network coverage, especially in underground tunnels. If your backend services aren’t publically available you can use a VPN client on your phone to allow you to test against local infrastructure.
The one thing you need to be aware of is other people on your bus/train stickybeaking if your app is commercially sensitive.
- Test WebView/bridged app HTML content independently of your app: we use a service that returns HTML5 content that is displayed and executed using a JavaScript bridge in both iOS and Android native apps. This content can be tested independently of the native apps which is both faster and easier than testing via the apps themselves.
- Fake back-end dependencies for quicker UI testing. Back-end dependencies of your application such as a database can be faked so you can quickly test your app’s UI is functioning as designed initially without having to worry about setting up an entire application stack for your testing.
- Test for accessibility on real devices: using iOS VoiceOver & Android TalkBack. These are both excellent screen readers which use very similar gestures to control the operating system and apps on your phone. The most common accessibility issues I discover are missing accessibility labels for UI controls and embedded WebViews not being marked accessible to the screen reader tools.
Summary
There we have it: eight tips for testing mobile apps. What lessons have you learned in the mobile testing space?
