Mobile Application Testing: Process, Tools and Techniques

The outbreak of smartphones and tablets forces us to be digitally available with speed. Keeping pace with communication tool developments, Lindiwe Vinson defines the methods used at Organic, Inc. where she leads her team discovering bugs using various key programs for both PC and Mac platforms.

The market for mobile applications increases every day and is becoming more and more demanding as technology grows. In a new study, Yankee Group predicts a $4.2 billion “Mobile App Gold Rush” by 2013 which includes:

  • Estimated number of smartphone users: 160 million
  • Estimated number of smartphone app downloads: 7 billion
  • Estimated revenue from smartphone app downloads: $4.2 billion

At Organic, our goal is to stay on the cutting edge of emerging platforms by launching new and diverse applications. We have this goal in mind when developing mobile web applications. We utilize some of the same styles of programming used for the developing of web applications. We also follow the same testing methodology employed for web development testing when testing our mobile applications.

1. Test Strategy is a high level document that defines “Testing Approach” to achieve testing objectives. The Test Strategy document is a static document meaning that it is not frequently updated. Components of the document include approach, risks, contingencies and recommendations, testing responsibility matrix, defect management process and resource requirements (schedule, tools, roles and responsibilities).

2. Performance Test Plan specifies how performance testing will proceed from a business perspective and technical perspective. At a minimum, a performance testing plan addresses dependencies and baseline assumptions, pre-performance testing actions, performance testing approach and performance testing activities.

3. Test Design Specification outlines, defines and details the approach taken to perform mobile application testing. The objective is to identify user flows and annotations, features to be tested, test scenarios, acceptance and release criteria.

4. Test Cases are derived from test scenarios and are identified in the test design specification. They are a set of test actions, test data/user input data, execution conditions, and expected results developed to verify successful and acceptable implementation of the application requirements.

5. Test Case Execution Summary Report provides information uncovered by the tests and is accomplished by the testing type. The report is used to relay the overall status of test execution on an iteration-by-iteration basis.

 

Although the mobile application testing process is basically the same we understand mobile devices have different peculiarities that must be kept in mind when deciding which testing types to use for authentication. The testing types used are predominantly unchanged but we do utilize different testing techniques and tools. Following are a list of testing types, techniques and tools used to support our mobile applications:

 

1. ADA Compliance Testing is used to measure and evaluate compliance to the Americans with Disabilities Act requirements. With mobile devices at an all-time high, there has been a surge of interest in developing applications that are in line with Mobile Web Best Practices (MWBP). To test accessibility we used the following tools and techniques.

¨ Create a URL test harness. The URL is checked via W3C mobileOK Checker, a free W3C service that validates the level of mobile-friendliness.

¨ The other test consists of using Apple’s Assistive Technology to test for screen magnification and VoiceOver for the blind and visually impaired.

 

2. Automated Testing is achieved using an emulator and a performance testing tool. The test runs on the device itself and is controlled by the PC. Results are captured using the performance testing tool. More details are provided below in the Performance Testing section.

¨ eggPlant is a QA automation and software testing product that allows you to emulate mobile devices and automate the testing. EggPlant can be downloaded for the Windows or Mac platforms.

 

3. Database Testing is very important for all applications. We check for data integrity and errors while editing, deleting and modifying the forms and all other DB related functionality. This testing is done manually, without the use of any testing tools.

 

4. Compatibility Testing assures the application works as intended with the selected device, operating system, screen size, display and internal hardware. Following are a list of tools that simulate different devices, operating systems, screens, etc.:

a. iPhoney is a free iPhone simulator powered by Safari (used on a MAC OS platform only).

b. Pad Peek allows you to see how your websites look when rendered on the iPad. This simulator is also free.

c. Adobe Device Central CS5 allows you to plan, preview, and test and deliver mobile applications. It is available with the Adobe Creative Suite® editions: Photoshop, Illustrator, Flash Professional, Dreamweaver After Effects and Fireworks.

d. DeviceAnywhere™ allows you to compose automated tests that run across multiple devices and multiple platforms/OS’s. DeviceAnywhere™ is a paid solution providing monthly and/or hourly options.

 

5. Functionality Testing includes the testing of controls, storage media handling options, and other operational aspects. Functionality testing for the mobile application is black-box testing and assures that the application functions per the business specifications. This testing is done manually.

 

6. Interoperability Testing includes testing of different functionalities within the iPad. For instance we uncovered that iTunes and Pandora end the play of music when launching the BroadFeed™. Interoperability testing had uncovered a major defect.

 

7. Mobile Analytics Testing is one of the most important tests and validates our ROI. We used Flurry™ to collect the analytics for BroadFeed™. To test correct implementation of analytics, we verified page and link tags, redirects, page source and user attributes as well as data capture.

a. Used Charles Web Debugging Proxy to verify the page and link tags, redirects requirements. This was achieved by changing the proxy settings in Charles then on the iPad; changed the Wi-Fi settings; “HTTP Proxy”, selected the Manual button and entered the desktop’s IP address.

b. Used the Flurry™ Dashboard to validate the data was captured correctly. The dashboard view provided us with snapshot of user metrics and usage.

c. Performance Testing is used to load and stress test the mobile application and database servers. To conduct performance testing we first created a test harness. Once this was created, we used Empirix eTester to record the script used to preform load and stress testing. Empirix eLoad Expert allowed us to easily and accurately test the performance and scalability of BroadFeed™ to ensure our customers would have the best possible experience. ELoad Expert simulated concurrent users, which allowed us to analyze the performance and identify any potential database issues.

 

8. Power Consumption Testing uncovers defects related to battery drainage caused by the application. Device settings can drain the battery life and this makes it hard to determine if the mobile application or the settings are the cause. The following are devices and the different testing methods for testing power consumption:

a. IPhone, iPod & iPad settings are adjusted; screen brightness, minimize use of location services, turn off push notifications, turn off other downloaded applications, fetch new data less frequently and turn off push mail. Then run the mobile application to determine the rate it took for the battery life to decrease. This testing is done manually without any testing tools.

b. Nokia Energy profiler is a stand-alone test and measurement application which allows monitoring the battery consumption on a target device.

 

9. Usability Testing is used to verify mobile interface, navigation, and intuitiveness of the application, as well as consistency, and soberness of color schemes.

A list of mobile device emulators used for testing:

 

Android emulator

Blackberry emulator

Dot Mobi emulator

Firefox Mobile emulator

Klondike WML emulator

LG emulator

Microsoft Devices emulator

Mozilla Fennec emulator

Nokia emulator

Opera Mini emulator

Opera Mobile emulator

Palm Pre / iPhone emulator

Samsung Java emulator

Samsung Platform emulator

Windows Mobile emulator

What we know for sure is that there will always be some level of manual testing when launching new applications, whether it is web or mobile. The solutions we use for testing combine manual testing, remote-manual testing, and a lot of testing using emulators and performance testing. We accomplish our testing goals utilizing an array of testing types to support the different techniques. We combine testing tools to help with the validation process. We try to remain cost effective by using freeware and in-house tools which allows us to conduct testing quickly and efficiently.

 

Lindiwe Vinson
Lindiwe Vinson is the Director of Technology at digital marketing agency Organic, Inc. and an expert in Quality Assurance. With more than twelve years of diverse experience, Lindiwe has a strong working knowledge SDLC, test planning, resourcing and contingency planning, metrics and defect reporting and performance testing; which includes an in-depth knowledge of engineering, testing, process management, change management and configuration management. She frequently contributes to Threeminds, Organic’s digital marketing blog. For more information, visit Organic.
Lindiwe Vinson
Lindiwe Vinson is the Director of Technology at digital marketing agency Organic, Inc. and an expert in Quality Assurance. With more than twelve years of diverse experience, Lindiwe has a strong working knowledge SDLC, test planning, resourcing and contingency planning, metrics and defect reporting and performance testing; which includes an in-depth knowledge of engineering, testing, process management, change management and configuration management.

The Related Post

Testing appears to be the least popular topic in Android development circles based on the relatively few books on Android app testing. Most tend to focus on development because, unfortunately (but true), application testing isn’t be something most developers think much about, or if they do, they don’t do it systematically (I’m guilty of this ...
Users aren’t likely to forgive and forget buggy apps. Mobile has big implications for business. The mobile experience is the customer experience, and you don’t get many second chances.
Removing the barriers to move and better mobile testing. Over the last decade, application testing has continually proved itself to be an important concern. When done well, testing can drastically reduce the number of bugs that make it into your release code (and thus actually affect your users). In addition, good testing approaches will help your ...
Organizations need to implement automated testing strategies designed specifically for mobile applications. Mobile device usage continues to gain momentum at increasing speed. Enterprises that delay mobile adoption face the danger of becoming competitively disadvantaged. But, before jumping in headlong, you need to be fully aware of the unique challenges that can arise when developing and implementing ...
Mobile is no longer an area that a few UX people specialize in, and we need to start designing and testing everything for smartphones and tablets as well as computers. If you’re new to mobile usability testing, fear not. It is not as hard as you might think but there are some key differences from ...
In today’s mobile-first world, a good app is important, meaning an effective Mobile Testing strategy is  essential.  
This article will cover 10 common mobile app testing mistakes to avoid when you are a software tester working in a mobile app testing and development environment. The 10 points may help you to start your mobile testing activities if you are new to mobile testing or they may help you to recap your existing mobile testing ...
To ensure the success of an app, QA must be involved in all stages of development. Quality Assurance (QA) plays a vital role in the development of mobile applications, but many overlook the critical nature of this piece of the app development process. To ensure the success of an app, QA must be involved in ...
I’ve spent the last six months or so testing mobile apps for both iOS and Android. Here’s eight of my key lessons learned: Automated UI testing tools for mobile apps are immature: whilst tools like WebDriver for automated UI testing of web apps are very mature, automated UI testing of native mobile apps is the ...
In the last issue on testing the SMAC stack we talked about the social and mobile aspects of testing. We will be referring to them in this article. In this issue part 2, we focus on the Analytics and Cloud aspect. The goal of this article is to understand a simple landscape of analytics and cloud.
Don’t make the mistake of assuming too many similarities. It is common knowledge that mobile applications don’t function in the same way as their web-based counterparts. The user experience is affected by a few other factors such as device and network capability. If you are building out a performance testing strategy for your mobile website ...
Gal Tunik explains the necessity of test automation in mobile applications, citing the unique challenges that that platform raises, including the intricate differences between mobile and desktop interfaces, and performance test results for applications as well as mobile networks. It is no surprise in this day and age of instant-on gratification that mobile devices are ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe