Devices matter. We don’t yet trust the mobile devices like we trust desktops and laptops. In the course of testing traditional web applications, rarely do you have to think about the model of the actual machine. In mobile, however, the behavior of an application can vary from device to device. You can no longer just think about browser/OS combinations.
Mobile emulators aren’t good enough. They get you part way there, but you still have to physically test on many devices.
The version of the OS can be quite important. Just as you think about the physical device, you also must consider how nuances of each OS version can affect application behavior: iOS 3, 4 or 5; Android 2.1, 2.2 , or 2.3; Blackberry 5, 6, or 7; Windows Mobile 7.0 or 7.5; etc. Add that to your test matrix.
Mobile web apps can become native apps. Tools like PhoneGap and Appcelerator enable web applications to be compiled into native applications. Now you might need to test a version of the application that runs in browsers and a version that runs natively.
Native phone features must now be considered. With the ability to turn web applications into native applications, this opens up access to native phone features, like the camera, accelerometer, address book, etc. These are not things we are all that familiar with.
Mobile web applications tend to be heavy on the JavaScript. Many factors lead to a thick-client architectural style for mobile applications. Business logic is pushed to the client-side—i.e. JavaScript —and the server-side becomes a set of services. Suffice to say that you now may find yourself much more concerned about how to test JavaScript.
Offline capability is important. This may become a first class concern. There are several reasons why a phone can lose connectivity, such as Airplane mode and driving into a tunnel, and mobile applications must be designed to deal with this fact—yet, another thing to add to your testing list.
Connection quality is unpredictable. Remember when you had to worry about dial-up users? It’s like that. Even if you assume your target user base will have at least 3G, connectivity can degrade or be lost at any time. One minute you’re on Wi-Fi, the next on 3G, and the next on Edge.
Data plans might be metered. Many users do not have unlimited data usage. Now the application architecture and design has to take into account how it will need to respect the user’s data plan. And you have to test it.
Instrumentation and test tool support is lagging on mobile devices. Any time there is emerging technology, the development of supporting tools (like testing tools) lags. For example, there are not currently a great many mature tool options available to automate tests of mobile applications across the spectrum of devices, OSes, browsers, etc.
The network provider might matter. Perhaps a particular network provider throttles connection speeds. Perhaps a particular network provider configures its phones in a particular way that might be important to know.
You have a lot to consider building your test matrix for testing mobile applications. The test strategies are still evolving. We are all still figuring out what matters in mobile application testing. Good luck!
John Roets
