Test automation provides great benefits to the software testing process and improves the quality of the results. It improves reliability while minimizing variability in the results, speeds up the process, increases test coverage, and ultimately can provide greater confidence in the quality of the software being tested. However, automation is not a silver bullet. It also brings some problems. The solution for test automation is to first define the test methodology, then choose the right enabling technology to help you implement the methodology. The chosen methodology should provide measurable improvements to the following four success drivers:
- Visibility
- Re-usability
- Scalability
- Maintainability
The four drivers above have a direct impact on three factors, manageability, productivity and cost efficiency, which ultimately lead to the tangible benefits that we are looking for a successful automation program to deliver.
Consider the following Figure 1. This diagram shows the results of a good test automation methodology. Good automation provides optimum productivity to the software testing effort. Hence, it leads to higher quality of the software releases. Test automation visibility provides measurability and control over the software development process, which keeps it manageable. With good visibility established, you can make effective management decisions about if, when, and how to do training and auditing to address the quality of tests. Reusability and scalability of test automation improves test productivity. However, productivity should be defined by (1) the quantity of tests (driven by reusability and scalability), and (2) the quality of tests (visibility into what the tests are actually doing helps improve the tests qualitatively). Quality and quantity are two different things. When test automation is reusable and scalable, the issue of quantity is resolved. When test automation is highly maintainable, the cost of ownership is minimized. Building test automation in a way that is more maintainable will lower the total cost of ownership of the test assets, making the overall testing effort more cost effective.
The quality of tests is mostly affected by the training and skills of the test staff. Automating a bad test does not improve its quality; it just makes it run faster. Good test design is a critical and often overlooked aspect of test automation. Test automation visibility by itself does not provide high quality tests. It merely enables us to have a view into how well the test designers are trained. Addressing the training issues will help in addressing test case quality issues.
Securing and developing competency is a secret to automation success.
Visibility, reusability, scalability, and maintainability lead to productivity and are the drivers for following benefits:
- Improved time-to-market.
- Improved quality of releases.
- Improved predictability.
- Improved Test/QA communication.
- Higher test coverage.
- Lower testing costs.
- Earlier detection of bugs.
- Lower technical support costs.
- More effective use of testing resources.
- Improved customer confidence and adoption.
After the test methodology and tools are set, the next step is to put the right people in place with the proper skills and training to do the work. The key to automation success is to focus your resources on the test production. That is to improve the quantity and quality of the tests, not to spend too many resources on automation production.
The most essential element to achieve these benefits is the methodology, not the tools.
In evaluating the return on investment with test automation, you need to look at the big picture. Think of the return-on-investment (ROI) equation, with the benefits on one side, and the costs on the other. For the benefits, consider the productivity, both in quantity and quality of tests. For the cost side of the equation, think about the reusability, scalability and maintainability of the tests in the context of the phases of a test automation effort:
- Deployment costs
- Test automation creation costs
- Execution costs
- Maintenance costs
Evaluate the ROI by considering if the costs are justified by:
- Faster/more tests?
- Faster/more test cycles?
- Better test coverage in each cycle?
- Higher quality of tests?
To make these evaluations, you need to have good visibility.
Often, the costs and benefits of test automation are uncertain due to lack of visibility. Management just does not know how much money it will spend on it, and how much benefit it will get from it. People are uncertain about how to quantify the ROI and how to set up a way to monitor it to see if they are on target. This is symptomatic to starting a project, and six months later something gets derailed.
Excellent visibility leads to effective management of test automation production.
Defining the test methodology before choosing the right enabling technology should provide measurable improvements in visibility, reusability, scalability, and maintainability yielding many tangible benefits from test automation efforts. To read more about this, see LogiGear’s book Global Software Test Automation: An Executive Snapshot of the Industry.
