Mobile Test Pyramid

Mobile testers need to take a different approach when it comes to Test Automation.

Anyone who is involved in software testing and software Test Automation should know the Test Automation pyramid introduced by Mike Cohn (www.mountaingoatsoftware.com/).

This article contains excerpts from my book Hands-On Mobile App Testing published with Pearson Education.

As you can see in the following image, the typical pyramid consists of three layers. At the bottom, there is the automated unit-testing layer, in the middle the automated integration testing layer and at the top there is the automated end-to-end testing layer (including the user interface tests). Each layer has a different size, indicating the number of tests that should be written within each stage. Manual testing is not part of the test pyramid, hence it is shown as a cloud for additional testing work.

But this pyramid is not applicable to mobile apps and mobile test automation. Mobile testing requires a totally different set of testing activities like movement, sensors, different devices and networks compared to other software like desktop or web applications. Lots of manual testing is required to be sure that a mobile app is working as expected in the different usage scenarios.

Mobile Test Automation tools are currently not as mature as their counterparts for web and desktop applications, which leads to a flipped Test Automation pyramid. As the tools become increasingly mature, this pyramid is likely to flip back again because the default Test Automation pyramid is based on a more stable foundation (see the pyramid image below).

The default pyramid therefore can’t be used as an indicator for Test Automation and Manual Testing in the mobile world. The flipped testing pyramid looks like this: In this version of the pyramid, the automated unit testing layer is the smallest one. This is the case because not

The flipped testing pyramid looks like this:

In this version of the pyramid, the automated unit testing layer is the smallest one. This is the case because not every unit or method of mobile apps can be tested in an isolated manner. In some cases, different APIs, layers or systems may need to be faked or mocked in order to get the small unit to work. This is also the case for every other software application, but in some cases mocking or faking other systems for mobile apps is much more complex. This is not efficient from a technical or economic point of view. However, it’s no excuse for not writing mobile unit tests at all. The main logic of an app can/must be tested at the unit level.

The next stage is the end-to-end Test Automation layer. Within this layer, the whole app is tested from a user perspective. It will be tested to make sure the whole system is working, from the app’s user interface through to the backend system via a wireless network, including integration testing with different libraries or APIs. The integration testing layer is therefore part of the end-to-end layer.

The biggest change in this pyramid is that manual testing is part of it. Mobile testing requires lots of Manual Testing, and this can’t be replaced by Test Automation or any other tools yet.

Nevertheless, mobile Test Automation is a really important topic and every mobile tester should be able to write automated regression tests that provide fast feedback about the current quality state of an app. Furthermore, Test Automation helps the team build a reliable and robust mobile app that makes the customers happy.

Mobile Test Pyramid

The flipped testing pyramid has no stable foundation and mobile testing requires lots of Manual Testing, which is why I created my own mobile test pyramid consisting of four layers including Manual and Automated steps. The biggest layer of the pyramid is Manual Testing and forms the strong foundation for every mobile app project, followed by end-to-end testing, beta testing and a top layer comprising unit testing. The grey parts of the pyramid indicate the automated steps and the white parts are the Manual Testing steps. The beta-testing layer is new to the pyramid but essential to every mobile app project. Keeping the high expectations of the mobile users in mind requires this layer to be part of every mobile project to get early feedback from your mobile customers. You can either use a crowd testing approach for your beta testing or you can ask your colleagues to beta test early versions of your app to provide important feedback.

Note: The End-2-End and Unit tests layers can also be swapped as well as the beta and End-2-End layer. The amount of End-2-End tests and Unit tests can differ from project to project and from app to app.

I used this mobile test pyramid in several projects and it helped me set up a reliable, effective and valuable testing process.

If you want to read more about mobile testing, please have a look at my book Hands-On Mobile App Testing which is available as a printed copy as well as an eBook. More details under: http://www.handsonmobileapptesting.com/ or in any other online or offline bookstore.

#HappyTesting!

Daniel Knott
Daniel is a mobile testing expert working as Senior Software Test Engineer at XING’s Android team. He started his software testing career in 2003 as a trainee at IBM Germany. After his time at IBM, Daniel studied computer science with a focus on software development and testing. Since 2009, Daniel has worked for companies such as Accenture, AOE and XING. In several agile development projects he tested web, desktop or mobile applications. However, mobile testing became his passion and since the beginning of 2011 he is working in the mobile development and testing industry. He worked and is working with several mobile test automation tools such as Robotium, Calabash for iOS/ Android, Espresso and Keep It Functional. With the help of this tools, he developed a fully automated testing environment for Android and iOS. Daniel likes to share his knowledge and therefore he started to share his experience on his blog at http://www.adventuresinqa.com as well as in several testing magazines. Daniel is a well-known mobile expert, a speaker at various conferences in Europe and since 2014, he is the author of the book “Hands-On Mobile App Testing“, more information about the book can be found at http://www.handsonmobileapptesting.com. Get in touch with Daniel:
– XING: Profile ( www.xing.com/profile/Daniel_Knott )
Daniel Knott on Twitter

The Related Post

One of my current responsibilities is to find ways to automate, as much as practical, the ‘testing’ of the user experience (UX) for complex web-based applications. In my view, full test automation of UX is impractical and probably unwise; however, we can use automation to find potential UX problems, or undesirable effects, even in rich, ...
From cross-device testing, to regression testing, to load testing, to data-driven testing, check out the types of testing that are suitable for Test Automation. Scene: Interior QA Department. Engineering is preparing for a final product launch with a deadline that is 12 weeks away. In 6 weeks, there will be a 1 week quality gate, ...
Learn how to leverage TestArchitect and Selenium for turnkey, Automated Web testing. TestArchitect lets you create, manage, and run web-based automated tests on different types of browsers—using either a WebDriver or non-WebDriver technique. In this article, we will explore employing WebDriver for testing a web-based application with TestArchitect. TestArchitect with WebDriver is a tool for automating ...
Has this ever happened to you: You’ve been testing for a while, perhaps building off of a branch, only to find out that, after all of this time, there is something big wrong. It’s a bad build and now you have to go backwards, fix something, and get a new build. Basically, you just wasted ...
Source: From I.M.Testy (BJ Rollison’s blog) I just finished reading Implementing Automated Software Testing by E.Dustin, T. Garrett, and B. Gauf and overall this is a good read providing some well thought out arguments for beginning an automation project, and provides strategic perspectives to manage a test automation project. The first chapter made several excellent ...
Test automation can provide great benefits to the software testing process and improve the quality of the results…. but its use must be justified and its methods effective. The reasons to automate software testing lie in the pitfalls of manual software testing… As we all know too well, the average manual software testing program:
An Overview of Four Methods for Systematic Test Design Strategy Many people test, but few people use the well-known black-box and white-box test design techniques. The technique most used, however, seems to be testing randomly chosen valid values, followed by error guessing, exploratory testing and the like. Could it be that the more systematic test ...
What is the Automation ROI ticker? The LogiGear Automation Return on Investment (ROI) ticker, the set of colored numbers that you see above the page, shows how much money we presumably save our customers over time by employing test automation as compared to doing those same tests manually, both at the design and execution level.
The growing complexity of the Human-Machine Interface (HMI) in cars offers traditional testers an opportunity to capitalize on their strengths. The human-machine interface (HMI) is nothing new. Any user interface including a graphical user interface (GUI) falls under the category of human-machine interface. HMI is more commonly being used to mean a view into the ...
People who know me and my work probably know my emphasis on good test design for successful test automation. I have written about this in “Key Success Factors for Keyword Driven Testing“. In the Action Based Testing (ABT) method that I have pioneered over the years it is an essential element for success. However, agreeing ...
“Happy About Global Software Test Automation: A Discussion of Software Testing for Executives” Author: Hung Q. Nguyen, Michael Hackett, and Brent K. Whitlock Publisher: Happy About (August 1, 2006) Finally, a testing book for executives!, November 17, 2006 By Scott Barber “Chief Technologist, PerfTestPlus” Happy About Global Software Test Automation: A Discussion of Software Testing ...
As I wrote in various articles, organization is one of the 3 key requisites for successful automated testing, the other two being test design and automation architecture.

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe