Mobile Testing Is Mission Critical

shutterstock_110616332Users aren’t likely to forgive and forget buggy apps.

Mobile has big implications for business. The mobile experience is the customer experience, and you don’t get many second chances.




  • 90% of users report they stopped using an app due to poor performance.
  • 86% have deleted or uninstalled at least one app because of performance problems.
  • 52% of users say that a bad mobile experience makes them less likely to engage with a company.

These stats highlight the importance of mobile testing and how much room for improvement there is. Being on the service side, we get a lot of inquiries for mobile testing. What is and isn’t surprising is that most of the focus is on cost, not so much on requirements or expertise. It’s not surprising from the standpoint that the main test objective for just about every organization is to maximize testing within a narrow budget. But it is surprising that, most of the time, clients’ test requirements are what we consider minimum acceptability levels, given the importance of the mobile user experience to the business. Whether you do the testing yourself or outsource it, how you approach testing has a big impact on the final outcome.

Mobile testing can mean testing either mobile-optimized websites or mobile apps, or both. Either can create an
extensive testing matrix. At last count there are at least 15 mobile browsers — an important factor with responsive web design. There are five mobile operating systems in the U.S. — iOS, Android, Windows OS, Blackberry and Symbian — with a smattering of others in different parts of the world. Then there’s the multitude of devices with vastly different screen sizes and resolutions.

The challenge is to make the complexity as manageable as possible. This can be done by rationally analyzing the
market for your mobile applications by focusing on the primary consumers of your mobile sites and apps. In most cases your primary consumers will be a very targeted demographic, and share a lot of similarities in preferences as well as device choices. Analyzing the target audience in this way should give you a distribution that comes close to the Pareto principal, where 80% of your audience uses just 20% of the devices (and browsers). Comscore Reports is a good source of information on device and OS statistics.

While there are quite a few variations of the Android operating system, the vast majority of software bugs show up on all platforms. That means that testing thoroughly on a few platforms will be more beneficial than doing limited testing on a wide variety of platforms.

A good understanding of these dynamics lets you organize by the devices, OSes and browsers that are most important. This allows you to develop a test plan that lets you quickly test and release the apps for your major market (high risk of UX), and then test the remainder (low risk of UX). Understanding the primary devices and operating systems that you absolutely have to test, and doing those first, is a way to make your testing as efficient as possible.


According to a study by Recon Analytics, the average American upgrades their device every 22 months, which means that you have to stay on top of what’s happening with devices. While the multiplicity of devices makes up a significant part of the cost of testing, there are options that make it possible to minimize the cost associated with them.

Cloud providers let you rent devices as needed, but it’s far from straightforward: you need to research the offerings to see what will work the best. Two terms to be familiar with are “public” and “private” clouds. The public option is one where you can check out a device (or devices), test on them, and then release them so that they become available for someone else. This can be fine for testing mobile-optimized websites, shutterstock_101358808and running app tests after a new OS release. The caveat is that you have to reinstall your environment every time you start a new session. With the private cloud option, only you have access to the specific devices during the time you contract for. This has the benefit of having to only set up the environment once. It is also usually the preferred way to go when it’s a new app under development, so you can be certain it won’t slip into the public domain before you’re ready. For companies looking for a low cost solution, the private cloud option may not fit the bill.

Device emulators and simulators

Device emulators and simulators (virtual devices) are a low cost option for mobile testing. While these will never completely take the place of testing on real devices, they are a good way to augment testing. For some types of testing, virtual devices are superior to real devices, as you can simulate situations — like low battery conditions and weak signal strength — that would be difficult or impossible to create using real devices. Virtual devices also make a lot of sense for testing the long-tail of older devices that you still want to support when apps are updated.

Emulators are available from the respective OS (Android, iOS, Windows Phone, Blackberry, etc.) vendor’s SDKs for free, and there are other sources as well. Sauce Labs offers an emulator cloud platform, if you don’t want to create your own environment. In addition, there are tools to augment the device emulators, like Fiddler, NetLimiter and Charles Proxy, which simulate GSM/GPRS networks to allow for testing of near real-world network conditions.

A well thought-out combination of emulators, actual devices and cloud testing can provide broad test coverage and a high degree of automation. Emulators are a good start for testing applications that are under development, to test base functionality and/or multiple languages, and identify major bugs. Next, targeted and thorough testing can be performed on actual devices.

Automation Tools

With the amount of testing that mobile demands, automation is essential. It’s the only way to really optimize testing efficiency with all the variables. There are a lot of tool choices for automation. Some are free, and others you have to pay for. If you go the cloud route, automation tools can be part of the package from a number of providers. The wide choice of available automation tools makes it difficult to select the most suitable ones for a project.

LogiGear’s TestArchitect Mobile Plus automation platform lets you create your own test environment with a
host/agent setup. Agents are downloaded onto the mobile devices (via Wi-Fi or USB tether) to be tested, and tests are run and managed from the host machine. This setup allows for up to 40 devices to be tested concurrently from a single host machine, which provides physical device coverage necessary for most test requirements. A significant advantage of TestArchitect is that a single suite of tests can be used without modification for both emulators and actual devices, which further optimizes testing.

User Feedback

Though it arrives late in the game, user feedback is ideal for understanding where the bugs are in an application; but it’s not always easy to come by, except for the customer who complains. It can be useful to add user concerns to the test checklist.

If the app has been recently released or updated, it is useful to analyze user reviews and feedback that have been posted on social networks and the app marketplace. Problems are often found and documented in these environments by users, and this information can help identify important shortcomings.

There are quite a few analytical tools out there that can give insight into bugs. Although most of these tools are developer-focused, Countly is one that can benefit testers. Countly is defined as an open-source, real-time (constantly updating) mobile analytics platform. What’s great about this is that it gives an idea of how users are experiencing your app, and where the exit points are. Looking at this data an give you a good idea of what additional types of tests may need to be run in order to improve the app for the next release.

With the growing importance of mobile as an essential channel, it seems clear that it’s easy to make a strong
argument that mobile testing must be as thorough as possible. Not putting resources into mobile testing is asking for trouble, but the good news is that mobile testing can be done in a way that is cost-effective.


Joe Luthy, Marketing Director

Joe has over 15 years product marketing experience in technology and Internet services for major companies including AT&T and Verisign.

TestArchitect for Visual Studio is a keyword authoring platform extension designed specifically to enhance coded UI test automation in Visual Studio 2012.

Joe Luthy
Joe has over 15 years product marketing experience in technology and Internet services for major companies including AT&T and Verisign.TestArchitect for Visual Studio is a keyword authoring platform extension designed specifically to enhance coded UI test automation in Visual Studio 2012.

The Related Post

Drawing from the Greek mythology of the lotus eaters, Anne-Marie Charrett warns testers to be weary of enjoying early success too soon upon finding high impact bugs.
  Mobile analytics experts Julian Harty and Antoine Aymer have teamed up to deliver a 161-page handbook designed to help you “enhance the quality, velocity, and efficiency of your mobile apps by integrating mobile analytics and mobile testing”.
Testing appears to be the least popular topic in Android development circles based on the relatively few books on Android app testing. Most tend to focus on development because, unfortunately (but true), application testing isn’t be something most developers think much about, or if they do, they don’t do it systematically (I’m guilty of this ...
Manual testing teams may not be able to test all the processes with each build Test automation of applications has been around for many years. There are many of us in the automated testing field that started very early in the test automation phase, but the introduction of mobile devices has brought on a new angle ...
What you need to know for testing in the new paradigm This two part article analyzes the impact of the Internet of Things (IoT) product development on traditional testing. Part one of this series starts with a wide view on the IoT, embedded systems and device development aspects of testing. Part two, to be published ...
LogiGear Magazine – November 2011 – Mobile Application Testing Issue
Will testers be among the first IT professionals to shift their toolset and workflows from desktops and laptops to tablets and smartphones? As I’m sure you already know, a monumental shift from desktop to mobile is upon us. Not only have consumer applications started leaving the desktop behind, but B2B applications are also starting their ...
LogiGear Magazine – September 2013 – Mobile Testing
To help testers gain an edge, here’s a list of free resources Mobile testing is making leaps and bounds of progress in the overall testing space. As this field is highly dynamic, a tester must constantly evolve and improvise his or her knowledge of mobile testing. To help software testers gain an edge, I have compiled the following list ...
LogiGear Magazine December 2012 – Mobile Test Automation  
Great mobile testing requires creativity to find problems that matter. I’d like to take you through the thought process of testers and discuss the types of things they consider when testing a mobile app. The intention here is to highlight their thought processes and to show the coverage and depth that testers often go to.

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news