8 Lessons Learned in Mobile App Testing

I’ve spent the last six months or so testing mobile apps for both iOS and Android. Here’s eight of my key lessons learned:

  1. Automated UI testing tools for mobile apps are immature: whilst tools like WebDriver for automated UI testing of web apps are very mature, automated UI testing of native mobile apps is the opposite. Whilst Appium shows some promise as a cross-platform UI testing tool for mobile apps, in my opinion, the maintenance overhead is still high as the tests are quite flaky. I recommend more thorough automated unit and integration testing, possibly supplemented with WebView tests (see below) and some exploratory manual app testing of the UI.

Münster, Germany - August 25, 2011: A close up of an Apple iPhone 4 screen showing various social media apps, including Skype, Google+, Facebook, Twitter, LinkedIn, Myspace, tumblr, foursquare, and Messenger.

  1. Base your mobile OS testing on projected usage, not current usage. Mobile hardware and OS upgrade cycles are shorter than desktop hardware, and if you look at trends and test against those, chances are you will have more realistic usage based testing.

This chart from mixpanel shows iOS7 adoption across a very large number of devices. You can see iOS6 is trending downwards as iOS7 is broadly adopted, so you could put less focus on testing iOS6.

  1. Test on real devices: I prefer to test on real devices like phones and tablets so that I can get a realistic feel for how the app runs. There’s also some things that can’t be tested on emulators/simulators like push notifications which require a device id, and Apple VoiceOver for accessibility testing which isn’t available in the simulators.

I have found that iPod Touch’s make excellent test iOS devices as they’re a lot cheaper than iPhones, and you can buy them refurbished on the Apple store, although if you want to test in the wild (see below), you’ll need a cellular iOS device. If you’re after an iOS6 specific test device you’re only option is to buy a 4th generation iPod touch refurbished as all new Apple iOS devices now come with iOS7 pre-installed.

It’s easy to find cheap Android pre-paid phones for testing, and you can still buy Android 2.3 phones with lower-res screens which is good for older version compatibility testing.

  1. Enable wireless distribution/installation of your test application locally: to enable quick installation of updates to your app on both iOS and Android.

For wireless iOS distribution this involves have a build machine (which has to be a Mac) compile an IPA file which is copied to a web-server, along with an XML manifest (PLIST) file that describes the app. You then have a simple web page with a itms-services link to the manifest, which can be opened in Safari on iOS, prompting the user to automatically install. If you sign your iOS app using an enterprise account it will run on an unlimited number of iOS devices without registering them as test devices. Full details are here.

For wireless Android distribution it is much easier. Simply have your build server compile an APK file and copy it to a web-server along with a page linking to the APK directly. Any Android phone set to allow installation of apps outside the Play Store will download the file and prompt to install it. Voila.

  1. Test in the wild: I like testing during my bus/train ride home to see how my app handles with intermittent cellular network coverage, especially in underground tunnels. If your backend services aren’t publically available you can use a VPN client on your phone to allow you to test against local infrastructure.

The one thing you need to be aware of is other people on your bus/train stickybeaking if your app is commercially sensitive.

  1. Test WebView/bridged app HTML content independently of your app: we use a service that returns HTML5 content that is displayed and executed using a JavaScript bridge in both iOS and Android native apps. This content can be tested independently of the native apps which is both faster and easier than testing via the apps themselves.
  1. Fake back-end dependencies for quicker UI testing. Back-end dependencies of your application such as a database can be faked so you can quickly test your app’s UI is functioning as designed initially without having to worry about setting up an entire application stack for your testing.
  1. Test for accessibility on real devices: using iOS VoiceOver & Android TalkBack. These are both excellent screen readers which use very similar gestures to control the operating system and apps on your phone. The most common accessibility issues I discover are missing accessibility labels for UI controls and embedded WebViews not being marked accessible to the screen reader tools.

Summary

There we have it: eight tips for testing mobile apps. What lessons have you learned in the mobile testing space?

 

Alister

Alister is an Excellence Wrangler for WordPress.com at Automatic. He has extensive experience in automated software testing and establishing quality engineering cultures in lean cross-functional software development teams. He lives in Brisbane, Australia with his wife and three sons, and writes a popular software testing blog at watirmelon.com.

Alister Scott
Alister is an Excellence Wrangler for Wordpress.com at Automatic. He has extensive experience in automated software testing and establishing quality engineering cultures in lean cross-functional software development teams. He lives in Brisbane, Australia with his wife and three sons, and writes a popular software testing blog at http://watirmelon.com.

The Related Post

  Mobile analytics experts Julian Harty and Antoine Aymer have teamed up to deliver a 161-page handbook designed to help you “enhance the quality, velocity, and efficiency of your mobile apps by integrating mobile analytics and mobile testing”.
Testing appears to be the least popular topic in Android development circles based on the relatively few books on Android app testing. Most tend to focus on development because, unfortunately (but true), application testing isn’t be something most developers think much about, or if they do, they don’t do it systematically (I’m guilty of this ...
What you need to know for testing in the new paradigm This two part article analyzes the impact of the Internet of Things (IoT) product development on traditional testing. Part one of this series starts with a wide view on the IoT, embedded systems and device development aspects of testing. Part two, to be published ...
Gal Tunik explains the necessity of test automation in mobile applications, citing the unique challenges that that platform raises, including the intricate differences between mobile and desktop interfaces, and performance test results for applications as well as mobile networks. It is no surprise in this day and age of instant-on gratification that mobile devices are ...
This is the second part of a two part article that analyzes the impact of product development for the internet of things (IoT) on software testing.  Part one of this article (LogiGear Magazine, Sept 2014) gave a wide view on the IoT, embedded systems, and the device development aspects of testing on these projects. This ...
By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation cost effectively. In a previous article we outlined the importance of understanding the mobile ecosystem and test design for planning and executing mobile testing. The focus of this article is about efficient mobile test ...
25% of Americans own a tablet. Up from 11% of U.S. adults in July of 2011 to 18% in January of 2012. – Pew Internet & American Life Project Nigeria has close to 100 million mobile phone lines, making it Africa’s largest telecoms market. – Nigerian Communications Commission Google plans to sell 200 million Android ...
Convergence of Social Media, Mobile, Analytics, & Cloud [SMAC] is one of the hottest trends these days. It is a major business agenda forcing organizations to rethink their strategies and increase technology investments in this direction.
What you need to know to get started Introduction to mobile application testing Gone are the days when the telephone was an appliance that sat in a corner and had to ring to get our attention, or a computer was a machine only few people used — these devices are now an extension of our ...
In today’s mobile-first world, a good app is important, meaning an effective Mobile Testing strategy is  essential.  
Steps that will enable you to identify the weaknesses of your new app, its vulnerabilities and strengths. So you’ve just finished developing a nifty, customisable app that can help farmers track their produce from source to market via their mobile phone. You’re elated and want to get started marketing it right away. Not to burst ...
Organizations need to implement automated testing strategies designed specifically for mobile applications. Mobile device usage continues to gain momentum at increasing speed. Enterprises that delay mobile adoption face the danger of becoming competitively disadvantaged. But, before jumping in headlong, you need to be fully aware of the unique challenges that can arise when developing and implementing ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe