Should Automated Acceptance Tests Use the GUI the Application Provides

Introduction

As a consultant and trainer, I am often asked by my clients and students how to deal with automated acceptance tests. One common question is whether automated acceptance tests should use the graphical user interface (GUI) provided by the application.

Consider the Context

Like all testing questions, this is a question that can only be answered well when we consider context – the missions and the givens of the project and the product. We do that by asking more questions.

  • What questions are we trying to ask and answer with automated tests?
  • To what extent has testing been done at lower levels of the application?
  • To what extent are we trying to confirm known behavior vs. trying to discover or investigate as-yet-unrecognized problems?
  • What tools do we have available? What tools will we have to obtain?
  • What is our team’s perceived level of experience and skills with the tools?
  • Has the application been developed for testability? Does it have an easily scriptable interface below the GUI? Does it provide log files?
  • Can we get changes to the product and support from the developers when the application is insufficiently testable (e.g. when HTML elements are missing id tags)?
  • How might we value speed, precision, and volume of tests? Is the GUI the place where we will obtain optimal automation value for those considerations?
  • To what extent do we need to model real users and observe what they observe? To what extent is automation capable of making such observations? What things will a conscious, conscientious human – a skilled tester – be likely to notice that automation might miss?
  • How might we create or use oracles that will allow the tool to help us to recognize a problem? (For more on oracles see Doug Hoffman’s three part article “Using Oracles in Testing and Test Automation” (1))
  • Is a need to vary the data that is being used in a productive way (systematically, pseudo-randomly, or randomly)?
  • How might we leverage GUI automation to drive the application quickly to a point where human testers can take over?
  • How might automated GUI tests drive us toward confirmation bias – that is, the tendency to design and execute tests that confirm existing beliefs about the product?
  • How might automated GUI tests lead us towards automation bias – the tendency to view results from automated processes as superior to human processes.

This might look like a long list, but you don’t have to answer these questions in great detail, nor do you have to get each one of them exactly right. A couple of moments of reflection on each one – a couple of minutes altogether – is likely rather than forgetting to consider it at all. If you are uncertain or stuck, a few more minutes of investigation or exploration could provide a huge return on investment. As you are in the middle of the project, continue to ask these questions periodically as a means of ensuring that the value of what you’re doing exceeds the cost.

Experience and Observation

In general, my experience and observation has been that as testing gets closer to the GUI, the cost of automating tests increases while the value derived from automated tests decreases.

Automated tests at the unit level tend to be:

  • Simpler to automate
  • Easier to comprehend and maintain
  • More subject to falsifiable assertions that machines can evaluate
  • Appropriately specific, where specificity matters
  • Revealing for problems that are easier to troubleshoot and debug
  • More immediately responsive to developers (since the developers tend to be the ones writing and running them)
  • Confirmatory, in a place where confirmation is more useful and important

Automated tests at the GUI level tend to be:

  • Complex and difficult to program
  • Hard to understand and maintain
  • Inadequate for recognizing problems that humans can easily identify
  • Overly specific in places where ambiguity can be tolerated by people
  • Revealing for problems that are more difficult to troubleshoot and debug
  • Less immediately responsive to developers
  • Confirmatory, at a level where investigation and discovery are more important

Now: I’ve said that this is a question that we can only answer well in context – but there is another question entirely: what do we mean by “acceptance tests”, and depending on what we mean, do we want to automate them at all? You might like to check “User Acceptance Testing – A Context-Driven Perspective” for some thoughts on that (2).

Michael Bolton

Michael Bolton provides training and consulting services in software testing and is a co-author with James Bach of Rapid Software Testing, a course and methodology on how to do testing more quickly, less expensively, and with excellent results. Contact Michael at mb@developsense.com.

Notes:

  1. Hoffman, Doug: “Using Oracles in Test Automation“. LogiGear Newsletter, 2006. Available free at:
    Using Oracles in Testing and Test Automation (Part 1 of 3);
    Using Oracles in Testing and Test Automation (Part 2 of 3);
    Using Oracles in Testing and Test Automation (Part 3 of 3).
  2. Bolton, Michael: “User Acceptance Testing – A Context-Driven Perspective”. Proceedings of the Pacific Northwest Software Quality Conference 2007, page 535.
Michael Bolton
Michael Bolton provides training and consulting services in software testing and is a co-author with James Bach of Rapid Software Testing, a course and methodology on how to do testing more quickly, less expensively, and with excellent results.

The Related Post

For those that are new to test automation, it can look like a daunting task to undertake For those who are new to Automation, it can look like a daunting task to undertake, but it only seems that way. If we unpack it and pinpoint the fundamentals, we can have the formula for the desired ...
In recent years, much attention has been paid to setting up Test Automation frameworks which are effective, easy to maintain, and allow the whole testing team to contribute to the testing effort. In doing so, we often leave out one of the most critical considerations of Test Automation: What do we do when the Test ...
One of my current responsibilities is to find ways to automate, as much as practical, the ‘testing’ of the user experience (UX) for complex web-based applications. In my view, full test automation of UX is impractical and probably unwise; however, we can use automation to find potential UX problems, or undesirable effects, even in rich, ...
September Issue 2019: Advancing Automation
Understanding the benefits and challenges of Automating ERP is critical. According to SAP, ERP (Enterprise Resource Planning) “is the core processes that are needed to run a company: finance, human resources, manufacturing, supply chain, services, procurement, and others. At its most basic level, ERP integrates these processes into a single system. But new ERP systems ...
Learn how to leverage TestArchitect and Selenium for turnkey, Automated Web testing. TestArchitect lets you create, manage, and run web-based automated tests on different types of browsers—using either a WebDriver or non-WebDriver technique. In this article, we will explore employing WebDriver for testing a web-based application with TestArchitect. TestArchitect with WebDriver is a tool for automating ...
Looking for a solution to test your voice apps across devices and platforms? Whether you’re new or experienced in testing voice apps such as Alexa skill or Google Home actions, this article will give you a holistic view of the challenges of executing software testing for voice-based apps. It also explores some of the basic ...
It can be complicated to automate model-based testing. Here’s how to employ action words to get the job done.
Many organizations rely on HP Quality Center to design test plans and track test results. TestArchitect’s Quality Center integration makes working with QC as easy as pie. TestArchitect (TA) is a three-in-one tool for Test Management, Test Development, and Test Automation. Users can create and manage test assets, execute tests, track and analyze test results, ...
Recently while teaching a workshop on Testing Dirty Systems, I uttered this “Randyism” off the top of my head, “Test automation is not automatic.” I realized immediately that I had just concisely stated the problem in making test automation a reality in many organizations. Most testers know that test automation is not automatic. (Wouldn’t it be great?) However, ...
What is Ethereum Smart Contract Testing? What are its challenges? If you’re new to Smart Contract Testing, this in-depth guide will prepare you on how to test smart contracts successfully. Blockchain stands out due to its enormous implications. Everyone has heard of it, but few people know what the ramifications are for testers or how ...
The guide for CUI Automated Testing strategies, including chatbot testing and voice app testing. In the Software Testing industry, trends come and go that shape the future of testing. From Automation in Agile, to the DevOps era we are now in, trends are what evolve and improve our testing processes and ideologies. Currently, many researchers ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe