The Ins & Outs of Mobile App Testing

6.2Sep-2014-04Removing the barriers to move and better mobile testing.

Over the last decade, application testing has continually proved itself to be an important concern. When done well, testing can drastically reduce the number of bugs that make it into your release code (and thus actually affect your users). In addition, good testing approaches will help your team catch bugs earlier in the development lifecycle – resulting in a savings of both time and money (not to mention reputation with your users). Code that has good test coverage enables you and your team to make changes and introduce new features to your app without the fear of it breaking existing functionality.

The word “Testing” is a large umbrella, and is usually better understood when you break it down to specific types of testing. For example:

  • Unit Testing – automated tests written by developers, with each test targeting a narrow slice of application behavior.
  • Functional & Acceptance Testing – typically performed by QA personnel or automated UI testing frameworks.
  • Performance Testing – often performed manually with profiling tools (for example – heap and CPU profiling tools), though many mobile app developers are moving towards integrating app analytics to gather this data from real usage as well.

That’s certainly not an exhaustive list of the types of testing, but enough to make an important point: mobile applications face several challenges when it comes to testing. Key among those challenges are:

  • Platform & Device Diversity
  • Immature Tooling
  • Lack of Awareness

If you opt to write native applications for each target platform, then any code-level testing (i.e. – Unit Tests) will not be transportable as you move from Objective-C (iOS) to Java (Android). In addition, any scripted UI-Automation testing tools may not work for multiple platforms (or at the very least require separate scripts for each platform). Hybrid solutions like PhoneGap, or cross-compiled solutions like Xamarin can offer a single approach to unit testing (given a single codebase for multiple platforms) – but do not always offer the same level of quality as native tooling when it comes to performance profiling. Despite the trade offs involved, I’ve found in my own experience that the biggest barrier to entry in mobile app testing is often a lack of awareness of what tools are availableThat is the barrier which I hope to address in this post.

Unit Testing

Unit testing for specific platforms or cross-platform tools is not difficult, and your options abound. Let’s look at a sample of some of these choices.

iOS

iOS developers who’ve been writing Objective-C for a while may be familiar with OCUnit, which shipped with XCode prior to the XCTest framework. It’s still supported in XCode 5, but the understanding is that new and future projects should focus on using XCTest.

Don’t let Apple’s sparse documentation on unit testing deter you from checking out the XCTest framework. If you’re running an OS X Server, you can also take advantage of the XCode service’s continuous integration features. As part of a continuous integration workflow, you can create “bots”, which can continually build and test your app.

Many developers prefer a Behavior-Driven-Development (BDD) style syntax for unit testing. If this describes you, be sure to check out Kiwi – a BDD style unit testing framework for iOS.

One other important mention is OCMock a mocking framework for iOS. Mocks are an indispensable part of writing adequate tests around your application’s behavior.

Android

JUnit is perhaps by far the most well known (and officially recommended by Google) testing framework for Android. The JUnit Android extensions allow you to mock Android components, but I’ve also seen quite a number of Android developers use JUnit with Mockito, another Android mocking framework.

Robolectric takes a different – and very interesting – approach by allowing you to run your Android unit tests in the normal JVM (Java Virtual Machine), without the need for an emulator. This enables your tests to not only run from within your IDE, but also as part of a continuous integration workflow.

Qt

Qt made the top 5 most used CPTs in 2013. If you’re building mobile applications with Qt, you’ll be happy to know about QTestLib, a unit testing framework built by Nokia. Based on my research, it appears that QTestLib can be integrated with a 3rd party continuous integration workflow – enabling very helpful testing automation.

PhoneGap/Apache Cordova

Web-based hybrid approaches to mobile apps can take advantage of a host of testing and mocking frameworks, not to mention scripted UI/acceptance testing tools as well (more on that in a moment). When it comes to unit testing JavaScript, three of the biggest names are QUnit, Mocha and Jasmine. I’ve personally used all three, with my favorite setup including Mocha and expect.js (which provides a BDD style test syntax). Mocking and “spy” frameworks abound in JavaScript as well, with Sinon.js and JsMockito among the more popular stand-alone mocking options.

Many PhoneGap developers take advantage of tools like PhantomJS – which is a “headless” (no UI) WebKit browser, with a JavaScript API. PhantomJS can be easily integrated into a continuous integration workflow to automatically run unit tests against your hybrid mobile application’s codebase.

Xamarin

Xamarin uses a customized version of NUnit (ported from JUnit), called NUnitLite which enables you to write unit tests against your Xamarin iOS & Android projects. For any shared codebase, you can use the unit testing framework of your choice.

Scripted UI Testing

Not every team can afford to hire an army of manual QA testers, despite how valuable that can be. Automated tooling can bridge the gap.

If you’re writing native iOS and Android apps, you’re in luck. Apple provides an “Automation instrument” that will automate UI tests against your iOS mobile application. The Android SDK provides the “uiautomator” library, described as “A Java library containing APIs to create customized functional UI tests, and an execution engine to automate and run the tests.” In addition to these, you can use third party tools.

Among the more interesting developments in mobile UI automated testing is the emergence of an open source project named Appium. Appium uses the WebDriver Json Wire Protocol to interact with iOS, Android and Firefox OS apps and gives you the choice of writing your UI tests in any WebDriver-compatible language (Java, Objective-C, JavaScript, PHP, Python, Ruby, C#, Clojure, Perl and others).

Performance & Profiling

Apple’s Instruments is one of the more impressive native toolsets I’ve seen recently. With Instruments, it’s possible to profile how your app executes, run stress tests, record and replay user actions, create custom instruments and a lot more. If you’re writing native iOS apps & not using Instruments, I recommend reading through the Quick Start to get up to speed.

With Android apps, you have several (albeit, lower-level) tools available: Systrace & Traceview. You can also use the Device Monitor to view memory usage based on logcat messages.

For hybrid mobile apps, you have a host of mature desktop browser tools (Chrome Developer Tools, Firefox/Firebug, etc.), which you can bring to bear on your app to profile CPU usage, memory, DOM manipulation and much more.

Many mobile developers have started taking advantage of third party analytics services such as Google Mobile Analytics, Countly, EQATEC, Flurry, Perfecto Mobile’s MobileCloud Monitoring and many others. The focus of these kind of analytics is usually more about how your app is actually used, user engagement, demographics, feature popularity, etc. However, it provides an opportunity to measure certain pieces of application performance from within real-world usage. While I wouldn’t recommend this being your first line-of-defense in performance testing, having the ability to track real world performance metrics can be a powerful tool in tuning your application to your users’ needs.

We’ve only scratched the surface of the various testing options available for mobile app development. What testing approaches & tools are you using when writing mobile apps? If you’re not currently testing your application, what are some factors that would change your mind?

 (This article was originally published on Developer Economics – www.DeveloperEconomics.com)

Jim Cowart

Jim Cowart is a developer focusing these days on JavaScript, HTML, CSS, Node.js and open source in general. He’s most passionate about connecting with other developers – learning, sharing and promoting patterns and ideas that will make development more productive, enjoyable and empowering. Jim works at LeanKit as their Front End Architect and lives in Chattanooga with his wife and three sons. You can follow him on twitter – @ifandelse

Social accounts are:

Jim Cowart
Jim Cowart is a developer focusing these days on JavaScript, HTML, CSS, Node.js and open source in general. He’s most passionate about connecting with other developers – learning, sharing and promoting patterns and ideas that will make development more productive, enjoyable and empowering. Jim works at LeanKit as their Front End Architect and lives in Chattanooga with his wife and three sons.

The Related Post

Mobile is no longer an area that a few UX people specialize in, and we need to start designing and testing everything for smartphones and tablets as well as computers. If you’re new to mobile usability testing, fear not. It is not as hard as you might think but there are some key differences from ...
Test engineers face a rapidly changing mobile application landscape, making mobile test automation a necessity. We know that mobile apps are becoming increasingly complex along with the technological advances in tablets and smartphones. Test engineers have to address multiple challenges while testing data-centric mobile apps, including the growing diversity of device features, platforms, and technologies. Fortunately, ...
By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation cost effectively. In a previous article we outlined the importance of understanding the mobile ecosystem and test design for planning and executing mobile testing. The focus of this article is about efficient mobile test ...
What you need to know for testing in the new paradigm This two part article analyzes the impact of the Internet of Things (IoT) product development on traditional testing. Part one of this series starts with a wide view on the IoT, embedded systems and device development aspects of testing. Part two, to be published ...
  Mobile analytics experts Julian Harty and Antoine Aymer have teamed up to deliver a 161-page handbook designed to help you “enhance the quality, velocity, and efficiency of your mobile apps by integrating mobile analytics and mobile testing”.
The outbreak of smartphones and tablets forces us to be digitally available with speed. Keeping pace with communication tool developments, Lindiwe Vinson defines the methods used at Organic, Inc. where she leads her team discovering bugs using various key programs for both PC and Mac platforms.
iOS culture, even in many large organizations with skilled engineers, is behind on up-to-date testing practices. Agile development has long been all the rage; indeed, in most modern development shops the great agile methodologies are old hat. If you come from a software background like Ruby on Rails, Python, or certain Java niches, you may–until ...
The most significant facet of mobile testing is understanding the mobile ecosystem. Mobile applications are growing in use and sophistication along with the speed of the networks and the increasing power of the devices. To be sure, mobile application testing can be done with minimal automation, but a high level of test automation will help ...
Drawing from the Greek mythology of the lotus eaters, Anne-Marie Charrett warns testers to be weary of enjoying early success too soon upon finding high impact bugs.
A sampling of some free, online, and easy-to-use mobile device emulators that can help get you started with testing. ScreenFly A free, customizable tool to test your website on any screen size, including desktops, tablets, televisions, and mobile phones.
Whether Or Not You Have a Mobile App You’re walking down the street. You see something interesting, and you want to know more about it. What do you do? Do you wait until you get home, open up your laptop, and type “google.com” into your search bar?
In today’s mobile-first world, a good app is important, meaning an effective Mobile Testing strategy is  essential.  

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe