The most significant facet of mobile testing is understanding the mobile ecosystem.
Mobile applications are growing in use and sophistication along with the speed of the networks and the increasing power of the devices. To be sure, mobile application testing can be done with minimal automation, but a high level of test automation will help assure the best quality mobile applications.
The most significant facet of mobile testing is to understand what goes on in the mobile ecosystem to be able to design tests appropriately. This sounds obvious, but due to the complex nature of the mobile ecosystem, it isn’t. The multiplicity of devices and operating systems alone make for a very challenging testing matrix, but layered on that are a variety of hardware operational nuances. These include what are best described as interruption events; things like incoming call and message notification, change in network speed, and power warning notices.
The risk of having a test work for a while and then inexplicably break is high in the absence of a significant emphasis on up-front test design and structure. For example when testing an application download function, the outcome of the test is dependent on the application, the device and the OS version. What seems as basic as automatically installing an app and launching it without manual intervention can be surprisingly complex.
Design Then Test
For any automation effort the primary key to achieving success will be the approach. As a rule of thumb test automation is successful when the “5% rules of test automation” can be achieved, which means that no more than 5% of test cases should be executed manually, with no more than 5% of testing resources applied to reach that level of automation. This is aggressive, but achievable. Having this as a goal puts the focus on effective testing, not on test execution or automation scripting. Establishing an aggressive goal will help drive efficiency throughout the entire test team.
Anyone who has been testing for any length of time knows there are many ways to get a short-term boost in testing efforts. However, to achieve lasting results it is necessary to have a good high level organization of tests and an automation tool that can effectively execute tests and increase the scale of testing without adding overhead. Ideally, the automation tool should incorporate an agile keyword-driven automation framework. The proper organization and the right tool will enable mobile test processes that include all of these:
- Effectiveness. It must be robust and aggressive enough to find the hidden bugs and problems
- Efficiency. It must minimize time, costs and utilization of resources
- Manageability. Managing testing and test automation can be even more challenging than managing software development
- And…not boring or repetitive. Bored testers are not sharp testers. Monotonous execution tasks should be done by automation, so testers can focus on more creative and ambitious test scenarios.
At LogiGear, our test teams are trained on the Action Based Testing method (ABT). ABT borrows from state-of-the practice object-oriented software programming that radically changed software development. The approach it promotes makes test design and development essentially one and the same. ABT focuses on the use of action keywords to design tests for automation and the modular test design concept it promotes integrates especially well with Agile practices. In application we have found this approach enables test teams to consistently attain the 5% goal of test automation.
Test Tools
Few testing organizations are exempt from the pressure to test faster. The typical response is to focus on selecting an easy-to-use tool and then create manual tests routines and automate them, or create tests based on an existing tool. Either way can be adequate for a few hundred test cases, but it is difficult to scale. Test sequences captured with record and playback are fast to create but costly to maintain. Lengthy test scripts usually contain steps that are likely to be repeated in other tests. Any application change that includes common steps will impact all tests and increase maintenance time and/or require creating new tests. The rapid rate of change in the mobile environment almost guarantees test requirements will change. And we all know that nothing saps productivity like recreating hundreds of tests.
The TestArchitect tool set was specifically designed to support ABT and employs a modular concept to streamline test design and management. The benefit of the modular approach is that when there is a change in an application, only the associated task(s) or function(s) in a module need to be changed, not the entire test. Take a secure logon operation as an example. With ABT a test of the secure logon would be created as an action with a keyword name. For every test requiring logon, the action would be included so the test could execute. If the logon action changed, say from a numeric entry to a motion sequence, one update to the logon action script would automatically update every test that required the logon script.
We have, and are currently working with a number of companies to improve their mobile testing efforts by incorporating ABT and TestArchitect. What follows are the test processes supported in TestArchitect that make mobile testing successful.
One Test, Multiple Targets
There are 3 dominant code bases for mobile devices: iOS, Android and Windows Mobile that most applications will be designed for. For every platform, you will likely need to test different hardware configurations, for example iPhone and iPad, Nexus S and Galaxy Tab, etc. This means there is a substantial opportunity for bugs to appear between different versions, and every time you make a code change there is risk of breaking something that used to work. Here it should be evident that creating and maintaining separate tests for each variation is not very practical.
The ideal solution is to create one set of tests that can be used for multiple devices and applications. This is what action keywords and modular test design enable you to do— by separating tests from the underlying code. Tests are created using actions that outline the task of the test. The automation tool assembles the test scripts from the actions, freeing test developers from tedious coding. The abstraction engine in TestArchitect translates the test cases into the appropriate code for the target device(s). There is some work required by automation engineers to map the appropriate interface, but this is a fairly minor task. The result is that 100 or 1000 test cases can be multiplied across a number of devices and OSs, with a fractional use of testing resources.
Mobile UI Testing
Mobile devices rely heavily on object oriented user interfaces for screens and controls. Routines are increasingly executed by taps, swipes and pinches and less on keypad inputs. Test cases that verify the flow and dialogs of the UI should be created and executed early in the test cycle to catch UI issues that could impact automation in the later stages. This will eliminate hours of investigative time to find the tests that need to be corrected.
There are two methods to test the UI actions–image based and object based testing. Object access to controls is a much more powerful and a maintainable option. It will expose virtually all properties of a screen or control to the test tool and also be visible to a developer. These properties can be used to identify a control or to verify the control values. For example with this approach the caption on a button will be visible as the “text” property of that button.
Object oriented testing isn’t impacted by UI dynamics such as changes in background, image size or orientation. Automating the testing requires access to the OEM provisioning files (Apple’s is ipa and Android is APK). TestArchitect installs its signature in the file to be able to interact with the object. The downside is that not all apps can access the provisioning files, for example Android security will not allow access to built-in applications.
Image-based automation will provide a true “end user approach” that verifies the tests sees the same image as the user. The test automation engine acquires the images from the on-screen display to perform the tasks required for the test. It doesn’t matter what the underlying technology is; if it is visible on the screen, image-based automation can interact with it.
Since it appears that image-based testing would work more often than object-based testing it begs the question, “Why not just use image-based testing”? The answer is that it is generally more sensitive to even minor changes in the UI or the application under test, and it is harder to let the same test run across different devices with different screen resolutions. In order to verify the app continues to work as expected when any of the parameters change, test cases with images for those specific parameters are needed. Here again, the modular test structure in TestArchitect makes it possible to minimize maintenance by creating a module just for the images. By just updating the test in the image module, all tests containing the images will be updated.
Testing Multiple Devices
Any good mobile test automation will include testing multiple devices simultaneously. This can involve running one test on many devices in parallel, and depending on the application it may require testing devices while interacting with each other. These types of tests can be run with agents or simulators, or a combination of both.
Mobile emulators can be essential for developing and testing a quality application. Emulators are good from the standpoint of running standard UI interactions and being able to do exploratory and benchmark testing economically. The technology enables developers to verify certain functionality that is not specific to any device, carrier or operating system, and for testing usability including data input, button use, etc. – all with the convenience of a full keyboard and mouse. However, emulators and simulators don’t operate in the real world and it is always recommended to test real devices in the native operating environment. Once satisfied the test parameters are correct, automation agents can execute tests on the actual devices (and run tests in parallel with emulators) for specific versions of devices and or OS so that testing effort can be maximized.
Cloud based services are also an option for testing multiple devices. The benefit is that these services have a wide range of devices and you pay only for use. The risk is uncertainty of the validity of the outcome. Many of the phones used in cloud based services have been jail-broken which can result in unintended consequences. This can be the case when an application relies on a specific carrier’s network security–what worked fine in the lab may not work when released to customers. Again, a good way to validate this is to test one or more actual devices.
There are device connectivity options as well. Devices can be connected to the host machine via USB and Wi-Fi. In most cases a USB connection is necessary to set up the automation agent, but once operational the tests can be driven over Wi-Fi. This has the benefit of testing the sensitivity of the application and device to changes in speed and or network signal strength–which users will encounter in the real world.
Summary
The demands for mobile test automation will continue to grow along with the increasing adoption of mobile devices. Due to the multiple OS standards and the multitude of device providers testers face a highly complex testing environment. By adopting the right techniques and methods and employing the appropriate tools, it is possible to tackle complex automation and be able to effectively scale as the need requires. It all starts with planning. Part of that planning should include learning and implementing new methods and tools that will improve efficiency. Those who are committed to long-term success in their field recognize this and embrace it.
