Mobile Testing Is Mission Critical

shutterstock_110616332Users aren’t likely to forgive and forget buggy apps.

Mobile has big implications for business. The mobile experience is the customer experience, and you don’t get many second chances.




  • 90% of users report they stopped using an app due to poor performance.
  • 86% have deleted or uninstalled at least one app because of performance problems.
  • 52% of users say that a bad mobile experience makes them less likely to engage with a company.

These stats highlight the importance of mobile testing and how much room for improvement there is. Being on the service side, we get a lot of inquiries for mobile testing. What is and isn’t surprising is that most of the focus is on cost, not so much on requirements or expertise. It’s not surprising from the standpoint that the main test objective for just about every organization is to maximize testing within a narrow budget. But it is surprising that, most of the time, clients’ test requirements are what we consider minimum acceptability levels, given the importance of the mobile user experience to the business. Whether you do the testing yourself or outsource it, how you approach testing has a big impact on the final outcome.

Mobile testing can mean testing either mobile-optimized websites or mobile apps, or both. Either can create an
extensive testing matrix. At last count there are at least 15 mobile browsers — an important factor with responsive web design. There are five mobile operating systems in the U.S. — iOS, Android, Windows OS, Blackberry and Symbian — with a smattering of others in different parts of the world. Then there’s the multitude of devices with vastly different screen sizes and resolutions.

The challenge is to make the complexity as manageable as possible. This can be done by rationally analyzing the
market for your mobile applications by focusing on the primary consumers of your mobile sites and apps. In most cases your primary consumers will be a very targeted demographic, and share a lot of similarities in preferences as well as device choices. Analyzing the target audience in this way should give you a distribution that comes close to the Pareto principal, where 80% of your audience uses just 20% of the devices (and browsers). Comscore Reports is a good source of information on device and OS statistics.

While there are quite a few variations of the Android operating system, the vast majority of software bugs show up on all platforms. That means that testing thoroughly on a few platforms will be more beneficial than doing limited testing on a wide variety of platforms.

A good understanding of these dynamics lets you organize by the devices, OSes and browsers that are most important. This allows you to develop a test plan that lets you quickly test and release the apps for your major market (high risk of UX), and then test the remainder (low risk of UX). Understanding the primary devices and operating systems that you absolutely have to test, and doing those first, is a way to make your testing as efficient as possible.


According to a study by Recon Analytics, the average American upgrades their device every 22 months, which means that you have to stay on top of what’s happening with devices. While the multiplicity of devices makes up a significant part of the cost of testing, there are options that make it possible to minimize the cost associated with them.

Cloud providers let you rent devices as needed, but it’s far from straightforward: you need to research the offerings to see what will work the best. Two terms to be familiar with are “public” and “private” clouds. The public option is one where you can check out a device (or devices), test on them, and then release them so that they become available for someone else. This can be fine for testing mobile-optimized websites, shutterstock_101358808and running app tests after a new OS release. The caveat is that you have to reinstall your environment every time you start a new session. With the private cloud option, only you have access to the specific devices during the time you contract for. This has the benefit of having to only set up the environment once. It is also usually the preferred way to go when it’s a new app under development, so you can be certain it won’t slip into the public domain before you’re ready. For companies looking for a low cost solution, the private cloud option may not fit the bill.

Device emulators and simulators

Device emulators and simulators (virtual devices) are a low cost option for mobile testing. While these will never completely take the place of testing on real devices, they are a good way to augment testing. For some types of testing, virtual devices are superior to real devices, as you can simulate situations — like low battery conditions and weak signal strength — that would be difficult or impossible to create using real devices. Virtual devices also make a lot of sense for testing the long-tail of older devices that you still want to support when apps are updated.

Emulators are available from the respective OS (Android, iOS, Windows Phone, Blackberry, etc.) vendor’s SDKs for free, and there are other sources as well. Sauce Labs offers an emulator cloud platform, if you don’t want to create your own environment. In addition, there are tools to augment the device emulators, like Fiddler, NetLimiter and Charles Proxy, which simulate GSM/GPRS networks to allow for testing of near real-world network conditions.

A well thought-out combination of emulators, actual devices and cloud testing can provide broad test coverage and a high degree of automation. Emulators are a good start for testing applications that are under development, to test base functionality and/or multiple languages, and identify major bugs. Next, targeted and thorough testing can be performed on actual devices.

Automation Tools

With the amount of testing that mobile demands, automation is essential. It’s the only way to really optimize testing efficiency with all the variables. There are a lot of tool choices for automation. Some are free, and others you have to pay for. If you go the cloud route, automation tools can be part of the package from a number of providers. The wide choice of available automation tools makes it difficult to select the most suitable ones for a project.

LogiGear’s TestArchitect Mobile Plus automation platform lets you create your own test environment with a
host/agent setup. Agents are downloaded onto the mobile devices (via Wi-Fi or USB tether) to be tested, and tests are run and managed from the host machine. This setup allows for up to 40 devices to be tested concurrently from a single host machine, which provides physical device coverage necessary for most test requirements. A significant advantage of TestArchitect is that a single suite of tests can be used without modification for both emulators and actual devices, which further optimizes testing.

User Feedback

Though it arrives late in the game, user feedback is ideal for understanding where the bugs are in an application; but it’s not always easy to come by, except for the customer who complains. It can be useful to add user concerns to the test checklist.

If the app has been recently released or updated, it is useful to analyze user reviews and feedback that have been posted on social networks and the app marketplace. Problems are often found and documented in these environments by users, and this information can help identify important shortcomings.

There are quite a few analytical tools out there that can give insight into bugs. Although most of these tools are developer-focused, Countly is one that can benefit testers. Countly is defined as an open-source, real-time (constantly updating) mobile analytics platform. What’s great about this is that it gives an idea of how users are experiencing your app, and where the exit points are. Looking at this data an give you a good idea of what additional types of tests may need to be run in order to improve the app for the next release.

With the growing importance of mobile as an essential channel, it seems clear that it’s easy to make a strong
argument that mobile testing must be as thorough as possible. Not putting resources into mobile testing is asking for trouble, but the good news is that mobile testing can be done in a way that is cost-effective.


Joe Luthy, Marketing Director

Joe has over 15 years product marketing experience in technology and Internet services for major companies including AT&T and Verisign.

TestArchitect for Visual Studio is a keyword authoring platform extension designed specifically to enhance coded UI test automation in Visual Studio 2012.

Joe Luthy
Joe has over 15 years product marketing experience in technology and Internet services for major companies including AT&T and Verisign.TestArchitect for Visual Studio is a keyword authoring platform extension designed specifically to enhance coded UI test automation in Visual Studio 2012.

The Related Post

Whether Or Not You Have a Mobile App You’re walking down the street. You see something interesting, and you want to know more about it. What do you do? Do you wait until you get home, open up your laptop, and type “” into your search bar?
Will testers be among the first IT professionals to shift their toolset and workflows from desktops and laptops to tablets and smartphones? As I’m sure you already know, a monumental shift from desktop to mobile is upon us. Not only have consumer applications started leaving the desktop behind, but B2B applications are also starting their ...
Steps that will enable you to identify the weaknesses of your new app, its vulnerabilities and strengths. So you’ve just finished developing a nifty, customisable app that can help farmers track their produce from source to market via their mobile phone. You’re elated and want to get started marketing it right away. Not to burst ...
To start with, we need a Test schedule. The same is created in the process of developing the Test plan. In this schedule, we have to estimate the time required for testing of the entire Data Warehouse system. There are different methodologies available to create a Test schedule. None of them are perfect because the ...
A fully integrated test automation platform for mobile testing Mobile application testing is taking on greater significance as just about everything done on the desktop is transitioning to mobile. It’s taken a while for mobile testing in the consumer space to be made much of a priority. Now that sensitive data is being passed back ...
LogiGear_Magazine_December 2016_Riding the New Software Testing Wave  
The outbreak of smartphones and tablets forces us to be digitally available with speed. Keeping pace with communication tool developments, Lindiwe Vinson defines the methods used at Organic, Inc. where she leads her team discovering bugs using various key programs for both PC and Mac platforms.
A sampling of some free, online, and easy-to-use mobile device emulators that can help get you started with testing. ScreenFly A free, customizable tool to test your website on any screen size, including desktops, tablets, televisions, and mobile phones.
Test engineers face a rapidly changing mobile application landscape, making mobile test automation a necessity. We know that mobile apps are becoming increasingly complex along with the technological advances in tablets and smartphones. Test engineers have to address multiple challenges while testing data-centric mobile apps, including the growing diversity of device features, platforms, and technologies. Fortunately, ...
Testing appears to be the least popular topic in Android development circles based on the relatively few books on Android app testing. Most tend to focus on development because, unfortunately (but true), application testing isn’t be something most developers think much about, or if they do, they don’t do it systematically (I’m guilty of this ...
To ensure the success of an app, QA must be involved in all stages of development. Quality Assurance (QA) plays a vital role in the development of mobile applications, but many overlook the critical nature of this piece of the app development process. To ensure the success of an app, QA must be involved in ...
I am not a big fan of concepts which moves industry standards to IT. I am rather a Agile and Scrum guy. Managing multiple projects at once and trying to set a highest quality standard is a challenge and this book shows how industrial language can be translated into software development. I do not think that it ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news