Users aren’t likely to forgive and forget buggy apps.
Mobile has big implications for business. The mobile experience is the customer experience, and you don’t get many second chances.
- 90% of users report they stopped using an app due to poor performance.
- 86% have deleted or uninstalled at least one app because of performance problems.
- 52% of users say that a bad mobile experience makes them less likely to engage with a company.
These stats highlight the importance of mobile testing and how much room for improvement there is. Being on the service side, we get a lot of inquiries for mobile testing. What is and isn’t surprising is that most of the focus is on cost, not so much on requirements or expertise. It’s not surprising from the standpoint that the main test objective for just about every organization is to maximize testing within a narrow budget. But it is surprising that, most of the time, clients’ test requirements are what we consider minimum acceptability levels, given the importance of the mobile user experience to the business. Whether you do the testing yourself or outsource it, how you approach testing has a big impact on the final outcome.
Mobile testing can mean testing either mobile-optimized websites or mobile apps, or both. Either can create an
extensive testing matrix. At last count there are at least 15 mobile browsers — an important factor with responsive web design. There are five mobile operating systems in the U.S. — iOS, Android, Windows OS, Blackberry and Symbian — with a smattering of others in different parts of the world. Then there’s the multitude of devices with vastly different screen sizes and resolutions.
The challenge is to make the complexity as manageable as possible. This can be done by rationally analyzing the
market for your mobile applications by focusing on the primary consumers of your mobile sites and apps. In most cases your primary consumers will be a very targeted demographic, and share a lot of similarities in preferences as well as device choices. Analyzing the target audience in this way should give you a distribution that comes close to the Pareto principal, where 80% of your audience uses just 20% of the devices (and browsers). Comscore Reports is a good source of information on device and OS statistics.
While there are quite a few variations of the Android operating system, the vast majority of software bugs show up on all platforms. That means that testing thoroughly on a few platforms will be more beneficial than doing limited testing on a wide variety of platforms.
A good understanding of these dynamics lets you organize by the devices, OSes and browsers that are most important. This allows you to develop a test plan that lets you quickly test and release the apps for your major market (high risk of UX), and then test the remainder (low risk of UX). Understanding the primary devices and operating systems that you absolutely have to test, and doing those first, is a way to make your testing as efficient as possible.
Devices
According to a study by Recon Analytics, the average American upgrades their device every 22 months, which means that you have to stay on top of what’s happening with devices. While the multiplicity of devices makes up a significant part of the cost of testing, there are options that make it possible to minimize the cost associated with them.
Cloud providers let you rent devices as needed, but it’s far from straightforward: you need to research the offerings to see what will work the best. Two terms to be familiar with are “public” and “private” clouds. The public option is one where you can check out a device (or devices), test on them, and then release them so that they become available for someone else. This can be fine for testing mobile-optimized websites, 
and running app tests after a new OS release. The caveat is that you have to reinstall your environment every time you start a new session. With the private cloud option, only you have access to the specific devices during the time you contract for. This has the benefit of having to only set up the environment once. It is also usually the preferred way to go when it’s a new app under development, so you can be certain it won’t slip into the public domain before you’re ready. For companies looking for a low cost solution, the private cloud option may not fit the bill.
Device emulators and simulators
Device emulators and simulators (virtual devices) are a low cost option for mobile testing. While these will never completely take the place of testing on real devices, they are a good way to augment testing. For some types of testing, virtual devices are superior to real devices, as you can simulate situations — like low battery conditions and weak signal strength — that would be difficult or impossible to create using real devices. Virtual devices also make a lot of sense for testing the long-tail of older devices that you still want to support when apps are updated.
Emulators are available from the respective OS (Android, iOS, Windows Phone, Blackberry, etc.) vendor’s SDKs for free, and there are other sources as well. Sauce Labs offers an emulator cloud platform, if you don’t want to create your own environment. In addition, there are tools to augment the device emulators, like Fiddler, NetLimiter and Charles Proxy, which simulate GSM/GPRS networks to allow for testing of near real-world network conditions.
A well thought-out combination of emulators, actual devices and cloud testing can provide broad test coverage and a high degree of automation. Emulators are a good start for testing applications that are under development, to test base functionality and/or multiple languages, and identify major bugs. Next, targeted and thorough testing can be performed on actual devices.
Automation Tools
With the amount of testing that mobile demands, automation is essential. It’s the only way to really optimize testing efficiency with all the variables. There are a lot of tool choices for automation. Some are free, and others you have to pay for. If you go the cloud route, automation tools can be part of the package from a number of providers. The wide choice of available automation tools makes it difficult to select the most suitable ones for a project.
LogiGear’s TestArchitect Mobile Plus automation platform lets you create your own test environment with a
host/agent setup. Agents are downloaded onto the mobile devices (via Wi-Fi or USB tether) to be tested, and tests are run and managed from the host machine. This setup allows for up to 40 devices to be tested concurrently from a single host machine, which provides physical device coverage necessary for most test requirements. A significant advantage of TestArchitect is that a single suite of tests can be used without modification for both emulators and actual devices, which further optimizes testing.
User Feedback
Though it arrives late in the game, user feedback is ideal for understanding where the bugs are in an application; but it’s not always easy to come by, except for the customer who complains. It can be useful to add user concerns to the test checklist.
If the app has been recently released or updated, it is useful to analyze user reviews and feedback that have been posted on social networks and the app marketplace. Problems are often found and documented in these environments by users, and this information can help identify important shortcomings.
There are quite a few analytical tools out there that can give insight into bugs. Although most of these tools are developer-focused, Countly is one that can benefit testers. Countly is defined as an open-source, real-time (constantly updating) mobile analytics platform. What’s great about this is that it gives an idea of how users are experiencing your app, and where the exit points are. Looking at this data an give you a good idea of what additional types of tests may need to be run in order to improve the app for the next release.
With the growing importance of mobile as an essential channel, it seems clear that it’s easy to make a strong
argument that mobile testing must be as thorough as possible. Not putting resources into mobile testing is asking for trouble, but the good news is that mobile testing can be done in a way that is cost-effective.
