Test Automation Is Not Automatic

Recently while teaching a workshop on Testing Dirty Systems, I uttered this “Randyism” off the top of my head, “Test automation is not automatic.” I realized immediately that I had just concisely stated the problem in making test automation a reality in many organizations.

Most testers know that test automation is not automatic. (Wouldn’t it be great?) However, management many times does not know or accept that reality.

There are some test tools, such as unit test tools, that are practically automatically applied. My remarks in this article are aimed at the capture/playback and scripting tools for test automation.

The issues are that:

  • Not every test can or should be automated.
  • For those tests that can be automated, it takes time and effort to build the automation.
  • For those tests that have been automated, the tests must be maintained.
  • It takes time to lean how to use a tool.
  • It takes effort and planning to implement a test automation framework.

None of these are automatic, even with the best of tools.

Not every test can or should be automated

Think about the things you test that are not very repeatable. Or, they may be prone to constant change. Perhaps the things you test are developed in a technology that has little or no tool support.

Then, there are tests such as user acceptance tests that need people’s evaluation to judge acceptance.

Some tests require creativity and adaptation to perform. You may have to make judgments during the test, which may be too complex to describe in a script. Test automation leverages the mundane testing to give more time and attention to the unique tests.

Your job is to identify the tests that can be automated. (They don’t come labeled!) Then, you must understand the nature of the test, such as the pre-requisites, the steps to perform it, the exceptions, and where to find the expected results. None of this is automatic. It’s all test design and test implementation.

For those tests that can be automated, it takes time and effort to build the automation

It’s one thing to automate a function, but another to design a good test of the function. That’s why capture/playback is so appealing, yet lacking. The issues are: What are you testing in the capture session? Then, how can you extend those tests to add value?

You have to apply approaches like data-driven testing and keyword-driven testing, which take time and effort to understand and implement. These are not “out of the box” deliverables.

For those tests that have been automated, the tests must be maintained

This has been one of the consistent issues in test automation. There are things you can do to ease the maintenance burden, but it doesn’t do away with the issue.

For example, modular and reusable test scripts are very helpful in reducing the number of tests that must be maintained. Still, you must maintain the scripts you have.

This means you must know when the application has changed, what the changes were, and create new tests for those changes. This implies the presence of configuration management and traceability.

It takes time to learn how to use a tool

Of course, the learning curve varies by tool, but the fact remains that a tool with sophisticated features will take some time to learn. The learning curve also varies by person. Some people with deep experience in automation may be able to learn very fast, but when you consider the wider deployment, most people will fall on the lower end of the experience scale.

The time also varies by the approach used to get training. Some people try self-learning which is noble, but typically takes longer than classroom training. Mentoring from an experienced person may be the best approach.

You must assess your organization’s skills and abilities to know if mentoring will help. The best mentor in the world can’t help the person who isn’t ready to learn. Is that another “Randyism?”

It takes effort, money and planning to implement a test automation framework

The framework can take a variety of forms, but the context here is the organizational framework which provides a way to efficiently build and control test automation. Tools are only one-third of the picture. You also need processes with trained and motivated people to make everything work together.

Out of the box, tools are just software. A process framework helps with reuse and the maintenance of test automation.

Frameworks aren’t automatic, either. They must be adapted to fit each situation, therefore they require time, effort and funding to design and implement.

Conclusion

This article is certainly not an in-depth treatment of this topic. Entire books and training courses have been written to address these and other aspects of test automation.

I hope this expands on my simple statement “Test automation is not automatic” to provoke your own thinking on this reality.

For over twenty years now, I have seen a wide variety of test tool companies promote their tools as being effort-free, script-less, or however they choose to position their products. The evidence shows these are often empty claims. Just compare the numbers of people who have been successful in using test automation tools to those who have given up using the tools. It’s about 25% successful to 75% unsuccessful in my research.

I hope this article is an encouragement to those who have tried to implement test automation as well as to those who haven’t. It’s important to go into these projects with your eyes open and expectations at a realistic level. Once you get past the idea that the tools do all the work, you can do the planning and other work needed to increase your chances of success.

Randall Rice
Randy Rice is a thought leading author, speaker and practitioner consultant in the field of software testing and software quality. Rice has worked with organizations worldwide to improve the quality of their information systems and optimize their testing processes.

The Related Post

This is part 2 of a 2-part article series; part 1 was featured in the September 2020 issue of the LogiGear Magazine, and you can check it out here. Part 1 discussed the mindset required for Agile, as well as explored the various quadrants of the Agile Testing Quadrants model. Part 2 will delve into ...
This book isn’t for everyone, but everyone can get some value out of it. What I mean by that rather confusing statement is that folks working in Agile environments will likely want to throw the book across the room while folks in more bureaucratic environments like CMMI or other waterfall environments will likely get a ...
LogiGear Magazine – September 2010
How lagging automotive design principles adversely affect final products. Cars are integrating more and more software with every model year. The ginormous screen introduced by Tesla in their flagship Model S a few years ago was seemingly unrivaled at the time. Nowadays, screens of this size are not only commonplace in vehicles such as the ...
I got some comments on my post “Test Everything all the Time” — most notably people commenting that it’s impossible to test “everything”. I can’t agree more. The intention of the post was to make the point that we need to be able to test “everything we can” all the time. That is, you should ...
In order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.
Test execution and utility tools that can make your job easier My first exposure to the necessity for testers to have an array of tools was from the groundbreaking article “Scripts on my Toolbelt” by Danny Faught. Danny laid out the ideal approach to any testing job, and it got me thinking “How can I ...
LogiGear Magazine – The Big Testing Issue – April 2012
Learn how to leverage TestArchitect and Selenium for turnkey, Automated Web testing. TestArchitect lets you create, manage, and run web-based automated tests on different types of browsers—using either a WebDriver or non-WebDriver technique. In this article, we will explore employing WebDriver for testing a web-based application with TestArchitect. TestArchitect with WebDriver is a tool for automating ...
As our world continues its digital transformation with excitement in the advancement and convergence of so many technologies- from AI, machine learning, big data and analytics, to device mesh connectivity, nor should we forget VR and AR- 2017 promises to be a year that further transforms the way we work, play and take care of ...
< Michael Hackett sat down with EA’s Stephen Copp to discuss the world of integrated test platforms.
LogiGear Magazine January Trends Issue 2017

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe