The Automation of Mobile Testing – Should We or Shouldn’t We . . .

6.3 -Sep-2014-03Manual testing teams may not be able to test all the processes with each build

Test automation of applications has been around for many years. There are many of us in the automated testing field that started very early in the test automation phase, but the introduction of mobile devices has brought on a new angle in test automation of applications. Mobile device testing introduces new technical challenges along with new processes and testing tools. But should we use test automation for mobile applications that are on mobile devices? Many of us have mobile devices and we get our applications from the stores on Android or iOS and we go about our business.

Quality, performance and business rules?

As we know, there are at least three kinds of applications for Mobile Devices; Native, Mobile Web, and Hybrid. Like any application, developers need to build it, and testers test it. Now, when we introduce methodology into the mix, we add challenges. This agile development process introduces tight timelines and tight delivery dates. Developers are challenged to get the business processes into the applications as quickly as possible. This adds multiple builds daily, sometimes hourly. In the desktop and standard web testing process, whether automated or manual, we would expect to regression test each build, right? How many testers would you need to do that when builds are being delivered so quickly? Can manual testers do that and keep up with what they have tested, along with making sure that the business processes have been tested each time?

When there is a time crunch, usually there will be decisions made to stick to the critical business processes or new functionality and leave it at that. It sounds great, but is that the best way to test and make sure a quality product has been delivered to production? Users are accustomed to application pages being rendered within a few seconds, which means response times need to be verified before going into production. Users are accustomed to the application functionality being consistent each and every build. If there is a new build, users are not expecting to lose functionality or any visual effects. Our eyes are accustomed to application pages looking great each and every build. How is a small test team going to do that? Honestly, a manual testing team will probably not be able to test all the processes each time and will revert to the 80/ 20 rule, or something close. Now, there are some teams out there that will claim to have an army of testers that they will call up and have the start testing the build. That is awesome, however, even that army of testers may take the common path or the happy path. They may not have all the business rules or the defects to look at to see what the challenges or critical areas of the app are. How do they report back to a central point and collaborate on what they have tested and what the results were? How would they also add requirements trace-ability into the process? Unless this is a seasoned group, that will be a challenge.

Where test automation comes in.

From wherever you are in the world, using test automation can make testing more complete and add requirements coverage, test coverage, trace-ability, defect tracking, and overall test reporting easier to manage. From one report, management can see what percentage of the application has been tested, what defects have been introduced and what defects have been fixed. With test automation, when a defect is found, it is very easy to recreate that defect and the steps for it. When manual testers test and find a defect, if the steps are complex enough, then chances are, the tester will not follow the same path and the defect may not be reproduced. That may be a big issue.

With test automation, test scripts can be run off hours if needed. Now with mobile device testing, that will add complexity because some environments and devices will need to be configured to run the test. That may or may not be able to be automated. Either way, the scripts can be run and tests can be run without manual intervention once set up. With test automation, when builds are done multiple times during the day, we can find the changes to the application much faster than a manual tester.

How fast can scripts do that?

Well, that would depend upon the environments being used, the complexity of the application under test, and the performance of the application.

For the most part, running automated scripts will always test more functionality than a manual tester. If there is a critical path that needs to be tested, the scripts can be run specifically for that and they will be executed the same each and every time. For many teams, cost comes into the picture. Most departments have budgets and test automation tools are usually pretty expensive. Once research is done, it will be found that there are many tools out there that can create scripts for a fairly inexpensive cost with even cheaper renewal costs. For the teams that do not have much experience with test automation, it would be better to try a small project with one of the less expensive tools. And remember, cheaper cost does not mean cheaper in the way the tool works, but it may mean that you don’t get all the bells and whistles. Compromises will always need to be added.

Should you use test automation for mobile device testing?

We can ask the same question for the test automation of a desktop application. It does not just need to be answered for mobile device testing. If your team plans to create subsequent updates to the app, or create the application in phases, I would highly recommend some sort of test script automation. If the plan is to just create a quick application one time and that is all, then it is up to the team to decide whether or not it is cost effective to automate the testing process. There will always be a benefit to test automation as a whole because test automation is not just script execution. Test automation is the whole testing process. That is where the real benefit for the whole project comes in. But for development and testers, script automation will help by allowing the manual testers to move onto some other critical tasks and leave the regression to the automation team.

Whether you have 4 or 400 testers, there will always be something that will be missed. Adding test automation will add test coverage and higher requirements coverage to your testing results. Mobile Device test automation will use a lot of the same processes that standard test automation uses. The tools may be different, but the rest will be similar.

(This article was originally published on


Sam Mullin

Sam Mullin is a specialist in Mobile Device Test Automation. He has contributed to test planning, design, and execution of automated testing efforts within the healthcare, electricity, and banking industries. He continues to work on providing specialized support for mobile device test automation using various mobile device testing tools. Currently, Sam is working for Olenick and Associates located in Chicago Illinois (

Sam Mullin
Sam Mullin is a specialist in Mobile Device Test Automation. He has contributed to test planning, design, and execution of automated testing efforts within the healthcare, electricity, and banking industries. He continues to work on providing specialized support for mobile device test automation using various mobile device testing tools. Currently, Sam is working for Olenick and Associates located in Chicago Illinois

The Related Post

Gal Tunik explains the necessity of test automation in mobile applications, citing the unique challenges that that platform raises, including the intricate differences between mobile and desktop interfaces, and performance test results for applications as well as mobile networks. It is no surprise in this day and age of instant-on gratification that mobile devices are ...
To start with, we need a Test schedule. The same is created in the process of developing the Test plan. In this schedule, we have to estimate the time required for testing of the entire Data Warehouse system. There are different methodologies available to create a Test schedule. None of them are perfect because the ...
I’ve spent the last six months or so testing mobile apps for both iOS and Android. Here’s eight of my key lessons learned: Automated UI testing tools for mobile apps are immature: whilst tools like WebDriver for automated UI testing of web apps are very mature, automated UI testing of native mobile apps is the ...
Don’t make the mistake of assuming too many similarities. It is common knowledge that mobile applications don’t function in the same way as their web-based counterparts. The user experience is affected by a few other factors such as device and network capability. If you are building out a performance testing strategy for your mobile website ...
LogiGear Magazine – September 2013 – Mobile Testing
Mobile is no longer an area that a few UX people specialize in, and we need to start designing and testing everything for smartphones and tablets as well as computers. If you’re new to mobile usability testing, fear not. It is not as hard as you might think but there are some key differences from ...
LogiGear Magazine December 2012 – Mobile Test Automation  
Great mobile testing requires creativity to find problems that matter. I’d like to take you through the thought process of testers and discuss the types of things they consider when testing a mobile app. The intention here is to highlight their thought processes and to show the coverage and depth that testers often go to.
iOS culture, even in many large organizations with skilled engineers, is behind on up-to-date testing practices. Agile development has long been all the rage; indeed, in most modern development shops the great agile methodologies are old hat. If you come from a software background like Ruby on Rails, Python, or certain Java niches, you may–until ...
By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation cost effectively. In a previous article we outlined the importance of understanding the mobile ecosystem and test design for planning and executing mobile testing. The focus of this article is about efficient mobile test ...
Drawing from the Greek mythology of the lotus eaters, Anne-Marie Charrett warns testers to be weary of enjoying early success too soon upon finding high impact bugs.
LogiGear Magazine – November 2011 – Mobile Application Testing Issue

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news