What Is Test Automation Return on Investment?

What is the Automation ROI ticker?

The LogiGear Automation Return on Investment (ROI) ticker, the set of colored numbers that you see above the page, shows how much money we presumably save our customers over time by employing test automation as compared to doing those same tests manually, both at the design and execution level.

We’ve segmented this page into three sections: one is to clarify our assumptions; then, to provide our audience some definitions, and finally to offer an example of our approach. We hope this helps clarify how we view test automation ROI.

Definitions:

What is a test case and how does LogiGear define this? A test case is a self-contained experiment — a sequence of operational and verifying actions, with a specific set of data, towards the application under test — with a defined outcome.

At LogiGear we reflect test cases as a series of actions or lines of actions. This is quite different from what most testers are used to.

In our ticker, a New Test Case is a test case that is created once during the day by our test engineering teams. We capture this number in every 24 hour cycle.

In our ticker, a Modified Test Case is a test case that has already been created, using the above criteria, but has been modified to accommodate new changes in the application. This might be a new verification point or a new navigation method. We might change that test case several times during the day, and we again, update this number in every 24 hour cycle.

How do we reflect our presumed savings?

The presumed Money Saved is calculated by comparing the total number of test cases that are developed and run automatically to the same number of tests that would be run and developed manually. As noted above in our assumption section, we make some assumptions in our calculation. One being the time it takes to develop and run our keyword based automation vs. the time it takes to develop and run a manual test case. We also factor in low cost offshore services, that maximize not only the resource cost, but the implementation costs.

Here is our simple formula reflecting costs:

As noted above – in our services organization, we generally realize a five (5) fold improvement in test execution and at least a two (2) fold improvement in designing and updating new and modified tests through our keyword method.


Assumptions and Clarifications:

The ticker reflects test cases that are developed and modified daily by our offshore service teams in Vietnam. LogiGear has been helping companies with software testing and test automation for many years. Over those years, we’ve compiled the number of tests we do for our clients as a matter of pride and internal tracking. In that time-frame we have developed over 8 million tests on 3000+ projects (yes, that’s million with an M!).

On to our numbers…

I’m sure we can all agree that executing a manual test takes more time. What we tried to do is average out the time it would normally take to run through a standard manual test case (with the assumption that these tests are required to be executed multiple iterations). We also looked at the time it would take to develop a manual test case, using the standard manual test case narrative approach.

LogiGear’s assumption is that it takes roughly 10 minutes to execute and document a standard test case manually and 6 minutes to write that same test. Again, these are averages – we’re sure some would agree or disagree with these figures.

In test automation, if your test cases are not running faster, then you are doing something wrong. Automation should allow you to execute your test cases at least 3 to 5 times faster than conventional manual testing or 2 minutes for each test in our assumptions. LogiGear also feels that our keyword test design approach, Action Based Testing, should allow testers to write and develop tests faster; this improvement in time should be at least 2 times faster, or 3 minutes for each test in our calculations.

Lastly, we factor in the cost of developing and running these test cases both manually and automatically. The rate we use to calculate is our costs to do this work offshore in our Vietnam Testing Facility, which provides low cost high expertise testing services. The rate we use as a comparison to reflect the savings, is doing this same work in the United States.

Here is a simple matrix of our data we used for our calculation:


An Example of our Approach:

Let’s use a test example of checking a website registration system. This system has a registration dialogue and we want to test it for different inputs. Your organization might write test cases in a classic test case narrative format, like the example below:

The above method is not very friendly to automation, nor is it very efficient test case design.

However, for keyword automation, not only is the test case designed for automation, but it’s also a very efficient way for documenting test.

Below, we took the same set of tests that we created above using a traditional narrative approach with steps and expected outcomes, and reflected them as keywords.

At LogiGear, our tests look like this (plus!! we then automate these keywords):

This is a very efficient way of designing tests and they are in format that allows for easy test automation.


We’re here to help you get the best return on your testing investment possible. You can find case studies on our client success with Action Based Testing and TestArchitect.

Contact us today to find out how LogiGear can save your company money.


Here are some Case Studies that have adopted LogiGear’s low cost test services and automation methods:

{faqslider tabs}TestingROI_Casestudy{/faqslider}

LogiGear Corporation
LogiGear Corporation provides global solutions for software testing, and offers public and corporate software testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast, cost-effective results. Since 1994, LogiGear has worked with Fortune 500 companies to early-stage start-ups in, creating unique solutions to meet their clients’ needs. With facilities in the US and Viet Nam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

The Related Post

How to do UI test automation with the fewest headaches I’m currently interviewing lots of teams that have implemented acceptance testing for my new book. A majority of those interviewed so far have at some point shot themselves in the foot with UI test automation. After speaking to several people who are about to do ...
When configured with a Python harness, TestArchitect can be used to automate testing on software for custom hardware Unlike other proprietary and open source tools, that are able to automate only desktop, or mobile, TestArchitect (TA Test) has the ability to test the software that runs on hardware in the following ways: 1. TA can ...
People who know me and my work probably know my emphasis on good test design for successful test automation. I have written about this in “Key Success Factors for Keyword Driven Testing“. In the Action Based Testing (ABT) method that I have pioneered over the years it is an essential element for success. However, agreeing ...
< Michael Hackett sat down with EA’s Stephen Copp to discuss the world of integrated test platforms.
We’ve scoured the internet to search for videos that provide a wealth of knowledge about Test Automation. We curated this short-list of videos that cover everything from the basics, to the more advanced, and why Test Automation should be part of part of any software development organization. Automation Testing Tutorial for Beginners This tutorial introduces ...
LogiGear Magazine – October 2010
TestArchitect TM is the name we have given to our automation toolset. It reflects the vision that automated testing requires a well-designed architectural plan allowing technical and non-technical elements to work fluidly in their capacity. It also addresses the continual missing link of all test automation tools of how to design tests. In TestArchitect the test ...
From automotive Software Testing standards, testing techniques, and process, this article is an in-depth guide for those looking to transfer their existing skills to this exciting industry. For the Software Car, autonomous driving gets most of the hype, but most overlook the fact that there is so much more to Software Testing for the automotive ...
This is part 2 of a 2-part article series; part 1 was featured in the September 2020 issue of the LogiGear Magazine, and you can check it out here. Part 1 discussed the mindset required for Agile, as well as explored the various quadrants of the Agile Testing Quadrants model. Part 2 will delve into ...
LogiGear Magazine – April 2013 – Test Automation
In order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe