Pushing the Boundaries of Test Automation: an Overview of How to Automate the UX with Heuristics

One of my current responsibilities is to find ways to automate, as much as practical, the ‘testing’ of the user experience (UX) for complex web-based applications. In my view, full test automation of UX is impractical and probably unwise; however, we can use automation to find potential UX problems, or undesirable effects, even in rich, complex applications. I, and others, am working to find ways to use automation to discover these various types of potential problems. Here’s an overview of some of the points I have made. I intend to extend and expand on my work in future posts.

In my experience, heuristic techniques are useful in helping identify potential issues. Various people have managed to create test automation that essentially automates different types of heuristics.

EXAMPLES OF PUSHING THE BOUNDARIES

  • Dynamic Usability / Accessibility Testing—See the following article I wrote that describes some of my work in the area. The code is available here. You’re welcome to use and experiment with it.
  • Fighting Layout Bugs—This is by Michael Tamm. He described the work in a public “tech talk” at Google’s Test Automation Conference (GTAC) in 2009. The link is available on his project’s homepage.
  • Crawljax—This is another open-source project which applies actions. It uses patterns to determine when to apply the actions. I’ve seen it used for significant, global web applications. There is a video online which describes some of that work.
  • BiDi Checker—This software helps to identify problems related to bi-directional content on web sites and web applications. It successfully finds and identifies a wide range of potential issues.

You might notice that all the examples I’ve provided are available as free open-source software (FOSS). I’ve learnt to value open source because it reduces the cost of experimentation and allows us to extend and modify the code, e.g. to add new heuristics relatively easily. (You still need to be able to write code, however the code is freely and immediately available.)

AUTOMATION IS (OFTEN) NECESSARY, BUT NOT SUFFICIENT

Automation and automated tests can be beguiling, and paradoxically increase the chances of missing critical problems if we chose to rely mainly, or even solely, on the automated tests. Even with state of the art (the best we can do across the industry) automated tests, I still believe we need to ask additional questions about the software being tested. Sadly, in my experience, most automated tests are poorly designed and implemented, which increases the likelihood of problems eluding the automated tests.

Here are two articles which describe some key concerns:

The first describes how people can be biased into over-reliance on automation. It is called “Beware of Automation Bias by M.L. Cummings, 2004. The article is available online.

The second helped me understand where testing helps us work out which questions to ask (of the software) and that we need to use a process to identify the relevant questions. The article is called “5 Orders of Ignorance,” by Phillip G Armour, CACM 2000.

 

Julian Harty

Julian has been working in technology since 1980 and over the years has held an eclectic collection
of roles and responsibilities: He was the first software test engineer at Google in Europe, the Tester at Large for eBay group, and has consulted and helped lots of companies and projects globally. He’s also been a company director for a mix of companies and startups. Currently, Julian combines commercial work, parttime Ph.D. studies, and helping with improving education, teaching and learning using low-cost mobile devices particularly for disadvantaged schools globally. He has authored several books, most recently the Mobile Analytics Playbook which can be downloaded for free at: http://www.themobileanalyticsplaybook.com/. You can find lots of his work, including opensource projects, online.

Julian Harty
Julian has been working in technology since 1980 and over the years has held an eclectic collection of roles and responsibilities: He was the first software test engineer at Google in Europe, the Tester at Large for eBay group, and has consulted and helped lots of companies and projects globally. He’s also been a company director for a mix of companies and startups. Currently, Julian combines commercial work, parttime Ph.D. studies, and helping with improving education, teaching and learning using low-cost mobile devices particularly for disadvantaged schools globally. He has authored several books, most recently the Mobile Analytics Playbook which can be downloaded for free at: http://www.themobileanalyticsplaybook.com/. You can find lots of his work, including opensource projects, online.

The Related Post

Introduction In many of the Test Automation projects that we are involved with using our Action-Based Testing methodology, management has expressed a need to relate tests and test results to system requirements. The underlying thought is that automation will create extra possibilities to control the level of compliance to requirements of the system under test. ...
I got some comments on my post “Test Everything all the Time” — most notably people commenting that it’s impossible to test “everything”. I can’t agree more. The intention of the post was to make the point that we need to be able to test “everything we can” all the time. That is, you should ...
Two dominant manual testing approaches to the software testing game are scripted and exploratory testing. In the test automation space, we have other approaches. I look at three main contexts for test automation: 1. Code context – e.g. unit testing. 2. System context – e.g. protocol or message level testing. 3. Social context – e.g. ...
The huge range of mobile devices used to browse the web now means testing a mobile website before delivery is critical.
With the new year just around the corner, here’s a look at the Test Automation trends that have the potential to dominate. DevOps is being relied upon more than ever. With there being strong Market Drivers for the adoption of DevOps, the need for Test Automation has also never been greater. But what’s next after ...
TestArchitect TM is the name we have given to our automation toolset. It reflects the vision that automated testing requires a well-designed architectural plan allowing technical and non-technical elements to work fluidly in their capacity. It also addresses the continual missing link of all test automation tools of how to design tests. In TestArchitect the test ...
As I wrote in various articles, organization is one of the 3 key requisites for successful automated testing, the other two being test design and automation architecture.
“Happy About Global Software Test Automation: A Discussion of Software Testing for Executives” Author: Hung Q. Nguyen, Michael Hackett, and Brent K. Whitlock Publisher: Happy About (August 1, 2006) Finally, a testing book for executives!, November 17, 2006 By Scott Barber “Chief Technologist, PerfTestPlus” Happy About Global Software Test Automation: A Discussion of Software Testing ...
LogiGear Magazine – April 2013 – Test Automation
LogiGear Magazine – March 2011 – The Agile Test Automation Issue
The success of Automation is often correlated to its ROI. Here are 5 KPIs that we find universally applicable when it comes to quanitfying your Test Automation.
*You can check the answer key here

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe