Mobile Testing Is Mission Critical

shutterstock_110616332Users aren’t likely to forgive and forget buggy apps.

Mobile has big implications for business. The mobile experience is the customer experience, and you don’t get many second chances.




  • 90% of users report they stopped using an app due to poor performance.
  • 86% have deleted or uninstalled at least one app because of performance problems.
  • 52% of users say that a bad mobile experience makes them less likely to engage with a company.

These stats highlight the importance of mobile testing and how much room for improvement there is. Being on the service side, we get a lot of inquiries for mobile testing. What is and isn’t surprising is that most of the focus is on cost, not so much on requirements or expertise. It’s not surprising from the standpoint that the main test objective for just about every organization is to maximize testing within a narrow budget. But it is surprising that, most of the time, clients’ test requirements are what we consider minimum acceptability levels, given the importance of the mobile user experience to the business. Whether you do the testing yourself or outsource it, how you approach testing has a big impact on the final outcome.

Mobile testing can mean testing either mobile-optimized websites or mobile apps, or both. Either can create an
extensive testing matrix. At last count there are at least 15 mobile browsers — an important factor with responsive web design. There are five mobile operating systems in the U.S. — iOS, Android, Windows OS, Blackberry and Symbian — with a smattering of others in different parts of the world. Then there’s the multitude of devices with vastly different screen sizes and resolutions.

The challenge is to make the complexity as manageable as possible. This can be done by rationally analyzing the
market for your mobile applications by focusing on the primary consumers of your mobile sites and apps. In most cases your primary consumers will be a very targeted demographic, and share a lot of similarities in preferences as well as device choices. Analyzing the target audience in this way should give you a distribution that comes close to the Pareto principal, where 80% of your audience uses just 20% of the devices (and browsers). Comscore Reports is a good source of information on device and OS statistics.

While there are quite a few variations of the Android operating system, the vast majority of software bugs show up on all platforms. That means that testing thoroughly on a few platforms will be more beneficial than doing limited testing on a wide variety of platforms.

A good understanding of these dynamics lets you organize by the devices, OSes and browsers that are most important. This allows you to develop a test plan that lets you quickly test and release the apps for your major market (high risk of UX), and then test the remainder (low risk of UX). Understanding the primary devices and operating systems that you absolutely have to test, and doing those first, is a way to make your testing as efficient as possible.


According to a study by Recon Analytics, the average American upgrades their device every 22 months, which means that you have to stay on top of what’s happening with devices. While the multiplicity of devices makes up a significant part of the cost of testing, there are options that make it possible to minimize the cost associated with them.

Cloud providers let you rent devices as needed, but it’s far from straightforward: you need to research the offerings to see what will work the best. Two terms to be familiar with are “public” and “private” clouds. The public option is one where you can check out a device (or devices), test on them, and then release them so that they become available for someone else. This can be fine for testing mobile-optimized websites, shutterstock_101358808and running app tests after a new OS release. The caveat is that you have to reinstall your environment every time you start a new session. With the private cloud option, only you have access to the specific devices during the time you contract for. This has the benefit of having to only set up the environment once. It is also usually the preferred way to go when it’s a new app under development, so you can be certain it won’t slip into the public domain before you’re ready. For companies looking for a low cost solution, the private cloud option may not fit the bill.

Device emulators and simulators

Device emulators and simulators (virtual devices) are a low cost option for mobile testing. While these will never completely take the place of testing on real devices, they are a good way to augment testing. For some types of testing, virtual devices are superior to real devices, as you can simulate situations — like low battery conditions and weak signal strength — that would be difficult or impossible to create using real devices. Virtual devices also make a lot of sense for testing the long-tail of older devices that you still want to support when apps are updated.

Emulators are available from the respective OS (Android, iOS, Windows Phone, Blackberry, etc.) vendor’s SDKs for free, and there are other sources as well. Sauce Labs offers an emulator cloud platform, if you don’t want to create your own environment. In addition, there are tools to augment the device emulators, like Fiddler, NetLimiter and Charles Proxy, which simulate GSM/GPRS networks to allow for testing of near real-world network conditions.

A well thought-out combination of emulators, actual devices and cloud testing can provide broad test coverage and a high degree of automation. Emulators are a good start for testing applications that are under development, to test base functionality and/or multiple languages, and identify major bugs. Next, targeted and thorough testing can be performed on actual devices.

Automation Tools

With the amount of testing that mobile demands, automation is essential. It’s the only way to really optimize testing efficiency with all the variables. There are a lot of tool choices for automation. Some are free, and others you have to pay for. If you go the cloud route, automation tools can be part of the package from a number of providers. The wide choice of available automation tools makes it difficult to select the most suitable ones for a project.

LogiGear’s TestArchitect Mobile Plus automation platform lets you create your own test environment with a
host/agent setup. Agents are downloaded onto the mobile devices (via Wi-Fi or USB tether) to be tested, and tests are run and managed from the host machine. This setup allows for up to 40 devices to be tested concurrently from a single host machine, which provides physical device coverage necessary for most test requirements. A significant advantage of TestArchitect is that a single suite of tests can be used without modification for both emulators and actual devices, which further optimizes testing.

User Feedback

Though it arrives late in the game, user feedback is ideal for understanding where the bugs are in an application; but it’s not always easy to come by, except for the customer who complains. It can be useful to add user concerns to the test checklist.

If the app has been recently released or updated, it is useful to analyze user reviews and feedback that have been posted on social networks and the app marketplace. Problems are often found and documented in these environments by users, and this information can help identify important shortcomings.

There are quite a few analytical tools out there that can give insight into bugs. Although most of these tools are developer-focused, Countly is one that can benefit testers. Countly is defined as an open-source, real-time (constantly updating) mobile analytics platform. What’s great about this is that it gives an idea of how users are experiencing your app, and where the exit points are. Looking at this data an give you a good idea of what additional types of tests may need to be run in order to improve the app for the next release.

With the growing importance of mobile as an essential channel, it seems clear that it’s easy to make a strong
argument that mobile testing must be as thorough as possible. Not putting resources into mobile testing is asking for trouble, but the good news is that mobile testing can be done in a way that is cost-effective.


Joe Luthy, Marketing Director

Joe has over 15 years product marketing experience in technology and Internet services for major companies including AT&T and Verisign.

TestArchitect for Visual Studio is a keyword authoring platform extension designed specifically to enhance coded UI test automation in Visual Studio 2012.

Joe Luthy
Joe has over 15 years product marketing experience in technology and Internet services for major companies including AT&T and Verisign.TestArchitect for Visual Studio is a keyword authoring platform extension designed specifically to enhance coded UI test automation in Visual Studio 2012.

The Related Post

To ensure the success of an app, QA must be involved in all stages of development. Quality Assurance (QA) plays a vital role in the development of mobile applications, but many overlook the critical nature of this piece of the app development process. To ensure the success of an app, QA must be involved in ...
Drawing from the Greek mythology of the lotus eaters, Anne-Marie Charrett warns testers to be weary of enjoying early success too soon upon finding high impact bugs.
I’ve spent the last six months or so testing mobile apps for both iOS and Android. Here’s eight of my key lessons learned: Automated UI testing tools for mobile apps are immature: whilst tools like WebDriver for automated UI testing of web apps are very mature, automated UI testing of native mobile apps is the ...
LogiGear Magazine – November 2011 – Mobile Application Testing Issue
What you need to know to get started Introduction to mobile application testing Gone are the days when the telephone was an appliance that sat in a corner and had to ring to get our attention, or a computer was a machine only few people used — these devices are now an extension of our ...
The outbreak of smartphones and tablets forces us to be digitally available with speed. Keeping pace with communication tool developments, Lindiwe Vinson defines the methods used at Organic, Inc. where she leads her team discovering bugs using various key programs for both PC and Mac platforms.
Here’s some good news: Jonathan Kohl has a new book out titled Tap Into Mobile Application Testing. This is a Leanpub release and is still in Beta, but the beauty of buying through Leanpub – you get all the author updates as they make them! However I will be keeping an eye out for the print-on-demand ...
Steps that will enable you to identify the weaknesses of your new app, its vulnerabilities and strengths. So you’ve just finished developing a nifty, customisable app that can help farmers track their produce from source to market via their mobile phone. You’re elated and want to get started marketing it right away. Not to burst ...
Devices matter. We don’t yet trust the mobile devices like we trust desktops and laptops. In the course of testing traditional web applications, rarely do you have to think about the model of the actual machine. In mobile, however, the behavior of an application can vary from device to device. You can no longer just ...
In the last issue on testing the SMAC stack we talked about the social and mobile aspects of testing. We will be referring to them in this article. In this issue part 2, we focus on the Analytics and Cloud aspect. The goal of this article is to understand a simple landscape of analytics and cloud.
This article will cover 10 common mobile app testing mistakes to avoid when you are a software tester working in a mobile app testing and development environment. The 10 points may help you to start your mobile testing activities if you are new to mobile testing or they may help you to recap your existing mobile testing ...
Mobile is no longer an area that a few UX people specialize in, and we need to start designing and testing everything for smartphones and tablets as well as computers. If you’re new to mobile usability testing, fear not. It is not as hard as you might think but there are some key differences from ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news