Strategize Your ’Mobile App Testing’ with ’M-Analytics’


Convergence of Social Media, Mobile, Analytics, & Cloud [SMAC] is one of the hottest trends these days. It is a major business agenda forcing organizations to rethink their strategies and increase technology investments in this direction.

What’s ‘Mobile Analytics’?

Mobile analytics involves measuring and analyzing data generated by mobile platforms and properties, such as mobile sites and mobile applications. It lets you track, measure and understand how your mobile users are interacting with your mobile sites and mobile apps. You can think of it as a superset of ‘Cookies’ which is used for Web traffic. With mobile analytics data, you can improve your cross-channel marketing initiatives, optimize the mobile experience for your customers, and grow mobile user engagement and retention.

Users statistics – number of Users, their characteristics, and where they come from

User behaviors – what actions your users are taking

Usage – Active users, sessions, session lengths, frequency, retention, etc.

Technical – Devices, carriers, firmware versions, errors

Collect comprehensive data for your mobile apps on all major operating systems – Installs, launches, average session length, and crashes

  • Visualize user navigation paths
  • Measure in-app payments and revenue
  • Geo-location analytics & targeting
  • Mobile engagement analysis
  • Mobile campaign analysis
  • Customized reports specific to your business App store statistics

Mobile Analytics & Testing

Testing services too are witnessing a growth in S-M-A-C with each evolving independently. In my earlier post “Benefits of Mobile App Testing in the Cloud” we looked at Mobile & Cloud integration for a better Mobile App Test Strategy. Objective of this post is to highlight the usage of ‘Analytics’ in ‘Mobile App Testing’!

The focal point of any Test strategy revolves around the customer (the Use-case). Wouldn’t it be better if as a Testing team you know your customer segment? App-usage workflow? Priority App features that customers use? What specific action caused the app to crash? Which features do users ignore, and why? Sounds good? Yeah! ‘Mobile Analytics’ can provide the answers to all these questions and more. Analytics can give you actionable insights into every aspect of your app resulting in an ‘Effective’ Mobile App Test Strategy.

You can only Test so much

Traditionally, Mobile App testing takes place before IT deploys an app, and in a controlled environment with a subset of users. But in the mobile era, it would be nearly impossible to test an app on every possible device and operating system combination a user could have. Consequently, the data IT gathers during mobile app testing doesn’t always reflect what will happen when an app goes into production.

There is no such thing as version one-and-done of a Mobile app. You’re always going to be iterating. That’s where Mobile Analytics comes in. Real-time data from Users who are using the app to perform real business tasks offers a more complete picture — and more actionable intelligence — than traditional App testing alone.


Structured Testing

A single bad experience can result in end users dumping the application. Using ‘Mobile Analytics’ in Mobile app testing, QA teams can adopt a structured testing approach focused on end-users. Structured tests help to proactively discover and remediate a broad number of potential application adoption issues – by exploring the app with a user mindset, seamlessly report back on the defects, provide feedback on the design, while slashing test cycle time. Leverage mobile analytics with actionable insights to improve your test effectiveness. Optimize your test cases and device testing matrix by continually measuring the user experience.

Customize Tests based on User Segments

Mobile Analytics helps you to see how different user segments interact with your App. Demographics, geographical location, platform, device used and time zone are all common examples of User segments that have different app usage patterns. Armed with this information, you can start tailoring your App Test Strategy for the appropriate segments to ensure a positive experience for all users.

Real-world Usability Tests

With ‘Mobile Analytics’ your App users effectively become testers. Individual profiles mean that you can inspect both one-night-stand installs and loyal users in order to determine what is successful and what is not. Get a high-level summary of how users interact with app content. Visualize the path users traveled from one Screen or Event to the next. This report can help you discover what app content keeps users engaged with your app. See user movement between Screens, Events, or a blended view of both Screens and Events. With this added information, Test teams can prioritize tests to focus on more relevant real-world usability tests.

A/B Testing Results

For most apps, the largest single one-day decrease in retention takes place on Day One; understanding the points at which users churn out and addressing them with A/B tests therefore has the most potential in absolute terms for retaining users in the crucial first moments after an app is launched.

The ability to analyze every aspect of your app during an A/B test is the most dynamic and informative way to ensure that you encourage optimal user engagement. What if you already know the ‘Grey’ area that has experienced a statistically significant change based on the test? This insight provides you with strong, math-driven A/B results quickly and easily. This understanding of behavioral analytics will also help you to identify where bottlenecks in the app might exist, which will have a profound effect on the kind of A/B testing you will do to address them.

Performance Tests

Mobile users have zero patience for poor performing mobile apps. Fix it or they’ll delete you. Test teams can leverage Mobile Analytics to analyze mobile app performance, stability, resource utilization, network latency and other factors to ensure an acceptable user experience.

Use App Speed reports to see how long different requests take to load in your app. Or Crashes & Exceptions report to identify the name and a brief description of the top Exceptions, or technical errors. You can define additional exception types (like Network failures and empty search results) in your app tracking code for more detail on other exceptions. Evaluate the performance of each screen in your app. Tracking scammers who simulate installs on some kind of data center with proxies and VPNs to tunnel out into a target country.

Not just Testing | It’s Quality Assurance

In technical landscape ‘Quality Assurance’ has replaced ‘Testing’. Now-a-days QA teams are not just responsible for reporting defects but to provide a deep App analysis and actionable insights with strategic recommendations to optimize App’s user experience – which features to spend resources on, and which flaws to fix or prioritize. User actions speak louder than words. Discover what works, and what doesn’t. Mobile Analytics allows data to be transformed first into insight and then into product improvements. Say in a location-based application, testing two different ways to interact with a map will likely increase use over time, thus impacting the entire life-cycle of the application.

For a Mobile QA team the recommended improvements to product features can be far more substantial when they are abstracted into best practices, made available to the entire organization, and implemented proactively in future development. Some improvements are so basic and broad – such as the results of fundamental A/B tests on pricing, tutorial flow, UI placement, etc. — that they can be assumed to be universally true. Establishing awareness of them throughout the organization can save untold time and effort in the future, allowing products to get to market faster and with higher quality.

Some real Examples

  1. Some XYZ App Users weren’t completing a workflow, and they were all abandoning the app at the exact same point — Recordings showed that users pinched and zoomed on their touchscreens at that point to more clearly see numbers they needed to enter. But there was a button they needed to press to confirm their entries on that screen, and it disappeared when they zoomed in, making it look like the app had frozen.
  2. For dating apps — Use of push notifications to tell the user that there’s something new, someone they can date or someone who wants to meet them etc. motivated them to open the app again.
  3. An app’s response timefor a certain action is a fraction of a second longer than it should be. Meanwhile, the analytics may also show that it takes twice as long as it should for a user to process an order in a customer relationship management app.
  4. Basic mobile app analytics report that a certain app crashes 10% of the time, typically about 15 seconds after it’s opened. In-app analytics reported that the crashes occur when users press the Add New Customer button on one of the app’s sub-menus. And the developer knew exactly which part of the code to fix.
  5. Why a user might have abandoned an app after only a few seconds. Perhaps the menu system is too confusing? The splash screen not impressive enough? The loading time too cumbersome?

Mobile Analytics give QA teams the information they need to determine whether products and campaigns are successful — and how to fix them if they’re not. They identify inefficiencies in Mobile App and potential solutions. With in-app analytics, QA can learn what causes apps to crash and what features users like. With this insight, QA team can streamline the app Test strategy, suggest improvements to back-end performance and identify bugs and poor design components.

Recognize that the data is important, but understand what it means. While raw data is important, it’s the organization and utilization of the data – the analytics – that really drives the push for mobile app companies to continuously strive for application optimization. A lean comprehensive overview of all of the aspects of your app in a quick and efficient manner helps you strategize the Testing solution.


Deepanshu Agarwal

Deepanshu Agarwal is a Software Testing professional, Team Lead and Consultant with a proven track record of 6.5+ years’ of strategic QA vision.

An accomplished and result-driven professional with proven ability to direct and improve quality programs, thereby ensuring effective & efficient project delivery. He started blogging at ‘Testing Mobile Apps‘ but later merged it into his other blog series at Software Testing Studio.

Deepanshu Agarwal
Deepanshu Agarwal is a Software Testing professional, Team Lead and Consultant with a proven track record of 6.5+ years’ of strategic QA vision. An accomplished and result-driven professional with proven ability to direct and improve quality programs, thereby ensuring effective & efficient project delivery. He started blogging at Testing Mobile Apps, but later merged it into his other blog series at Software Testing Studio.

The Related Post

By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation cost effectively. In a previous article we outlined the importance of understanding the mobile ecosystem and test design for planning and executing mobile testing. The focus of this article is about efficient mobile test ...
Will testers be among the first IT professionals to shift their toolset and workflows from desktops and laptops to tablets and smartphones? As I’m sure you already know, a monumental shift from desktop to mobile is upon us. Not only have consumer applications started leaving the desktop behind, but B2B applications are also starting their ...
Users aren’t likely to forgive and forget buggy apps. Mobile has big implications for business. The mobile experience is the customer experience, and you don’t get many second chances.
Removing the barriers to move and better mobile testing. Over the last decade, application testing has continually proved itself to be an important concern. When done well, testing can drastically reduce the number of bugs that make it into your release code (and thus actually affect your users). In addition, good testing approaches will help your ...
Strategies to Approach Mobile Web App Testing Mobile web technology has been continuously changing over the past few years, making “keeping up” challenging. In this article, Raj Subramanian covers the latest trends and changes happening in the mobile web and how testers can prepare for them.
This article will cover 10 common mobile app testing mistakes to avoid when you are a software tester working in a mobile app testing and development environment. The 10 points may help you to start your mobile testing activities if you are new to mobile testing or they may help you to recap your existing mobile testing ...
A sampling of some free, online, and easy-to-use mobile device emulators that can help get you started with testing. ScreenFly A free, customizable tool to test your website on any screen size, including desktops, tablets, televisions, and mobile phones.
CEO and founder of mVerify Corporation, Robert V. Binder tackles questions from field testers regarding such issues as strategic considerations when dealing with single stack apps versus globalized enterprise mobile apps, and methods and tools that developers and testers should be aware of. He also offers his own advice from lessons learned from experience. 1. ...
LogiGear Magazine, September 2015: Mobile Testing
Manual testing teams may not be able to test all the processes with each build Test automation of applications has been around for many years. There are many of us in the automated testing field that started very early in the test automation phase, but the introduction of mobile devices has brought on a new angle ...
Great mobile testing requires creativity to find problems that matter. I’d like to take you through the thought process of testers and discuss the types of things they consider when testing a mobile app. The intention here is to highlight their thought processes and to show the coverage and depth that testers often go to.
This is the second part of a two part article that analyzes the impact of product development for the internet of things (IoT) on software testing.  Part one of this article (LogiGear Magazine, Sept 2014) gave a wide view on the IoT, embedded systems, and the device development aspects of testing on these projects. This ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news