By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation cost effectively.
In a previous article we outlined the importance of understanding the mobile ecosystem and test design for planning and executing mobile testing. The focus of this article is about efficient mobile test execution and test coverage maximization.
Despite the growing number of mobile testing solutions the market is far from mature, which is one of the reasons why the majority of mobile device testing is still done manually. The primary issues with manual testing for mobile are scalability and cost. Outsourcing is always an option to increase the scale of testing, and then there is crowdsourcing. Crowdsourced mobile testing is being promoted as an ultra low-cost method of outsourced mobile testing. Crowdsourcing proponents tout flexibility, multilingual testing and a broad cross section of devices, but crowdsourcing is performed by a collection of freelancers so keep in mind the oft-used adverting disclaimer: “Results may vary.”
To adequately perform testing that assures mobile applications will meet customer expectations, test automation is necessary, and the need will only grow as mobile applications increase in complexity and importance. A high degree of test coverage can be achieved using both emulators and actual devices. How much testing to perform using emulators versus actual devices will depend on the budget, the automation tool, the application and the type of testing.
We have approached mobile test automation to make it possible for testers to employ a combination of testing assets—emulator/device/cloud testing resources—to efficiently achieve testing goals. Our automation tools provide the features and functionality to scale mobile test automation along with project needs and resources.
Emulators are ideal for testing new functionality or a new component design, and have some advantages over using actual devices. Logging faults and capturing screenshots are much simpler when working from a desktop, and you can simulate conditions that are hard to duplicate on real devices, like low battery power. A nice feature of TestArchitect is the ability to perform multilingual testing with emulators. This feature allows running identical tests in multiple languages in a short period of time.
Emulators can be accessed from the respective OS (Android, iOS, Windows Phone, Blackberry etc.) vendor’s SDK for free. Unlike iOS, Android isn’t the same for every device. For the best device approximation emulators from each Android device provider should be used. Then there are tools to augment the device emulators like Fiddler, NetLimiter and Charles Proxy that simulate GSM/GPRS networks to include testing of near real-world network conditions.
The main thing to keep in mind is that emulators and simulators only provide an approximation of device performance. From a QA perspective, emulators and simulators are fine for testing base functionality, but critical application sign-off should be done by live testing.
Testing a representative sample of actual devices is always ideal. The choice here is acquiring and updating an inventory of devices or using cloud base services. For overall maximum test coverage a public cloud option will provide broadest device selection. There are a number of cloud based options, each with their pros and cons. The big considerations are avoiding the need to recreate tests and security of the devices.
The accepted norm is to perform tests on 30 to 40 different devices currently available in the market. In cases where it’s essential to test actual devices, but less than the ideal number, applying Pareto’s Principle—80% of your customers will use just 20% of the devices available—should enable you to select the minimum number of devices that will provide the maximum test coverage.
Our approach to mobile cloud test automation was to create a host/agent set-up. Agents are downloaded onto the mobile devices (via wi-fi) to be tested, and tests are run remotely from the host machine. TestArchitect allows up to 40 devices to be tested simultaneously on a host or agent equipped machine to test within accepted norms. This approach eliminates API programming and provides the potential for regional testing—important for testing apps/devices on various operators’ networks, and in multiple languages—to validate applications for use on a global scale. The additional advantage of using TestArchitect is that tests used for any initial emulator testing can be run without modification on the actual devices, whether in the cloud or physically connected.
A well thought out combination of emulators, actual devices and cloud testing can provide broad test coverage and a high degree of automation. New applications can first be tested using emulators to test base functionality and/or multiple languages and identify major bugs. Next, targeted and thorough testing can be performed on actual devices.
Segregating testing in this manner can speed testing, and significantly help separate OS issues from device issues to minimize debugging time. Similarly, current applications can be tested on emulators of new or upgraded OS’s first, and then on a representative sample of next generation devices. Regression testing can be performed in the cloud across a wide range of devices with various hardware and OS versions.
Mobile devices are quickly becoming preferred to the desktop. To assure the best user experience, it is important to test mobile apps and websites with actual devices, and to keep this up requires automation. Mobile test automation is easily achievable. By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation and to do it cost effectively.
