What Is Test Automation Return on Investment?

What is the Automation ROI ticker?

The LogiGear Automation Return on Investment (ROI) ticker, the set of colored numbers that you see above the page, shows how much money we presumably save our customers over time by employing test automation as compared to doing those same tests manually, both at the design and execution level.

We’ve segmented this page into three sections: one is to clarify our assumptions; then, to provide our audience some definitions, and finally to offer an example of our approach. We hope this helps clarify how we view test automation ROI.

Definitions:

What is a test case and how does LogiGear define this? A test case is a self-contained experiment — a sequence of operational and verifying actions, with a specific set of data, towards the application under test — with a defined outcome.

At LogiGear we reflect test cases as a series of actions or lines of actions. This is quite different from what most testers are used to.

In our ticker, a New Test Case is a test case that is created once during the day by our test engineering teams. We capture this number in every 24 hour cycle.

In our ticker, a Modified Test Case is a test case that has already been created, using the above criteria, but has been modified to accommodate new changes in the application. This might be a new verification point or a new navigation method. We might change that test case several times during the day, and we again, update this number in every 24 hour cycle.

How do we reflect our presumed savings?

The presumed Money Saved is calculated by comparing the total number of test cases that are developed and run automatically to the same number of tests that would be run and developed manually. As noted above in our assumption section, we make some assumptions in our calculation. One being the time it takes to develop and run our keyword based automation vs. the time it takes to develop and run a manual test case. We also factor in low cost offshore services, that maximize not only the resource cost, but the implementation costs.

Here is our simple formula reflecting costs:

As noted above – in our services organization, we generally realize a five (5) fold improvement in test execution and at least a two (2) fold improvement in designing and updating new and modified tests through our keyword method.


Assumptions and Clarifications:

The ticker reflects test cases that are developed and modified daily by our offshore service teams in Vietnam. LogiGear has been helping companies with software testing and test automation for many years. Over those years, we’ve compiled the number of tests we do for our clients as a matter of pride and internal tracking. In that time-frame we have developed over 8 million tests on 3000+ projects (yes, that’s million with an M!).

On to our numbers…

I’m sure we can all agree that executing a manual test takes more time. What we tried to do is average out the time it would normally take to run through a standard manual test case (with the assumption that these tests are required to be executed multiple iterations). We also looked at the time it would take to develop a manual test case, using the standard manual test case narrative approach.

LogiGear’s assumption is that it takes roughly 10 minutes to execute and document a standard test case manually and 6 minutes to write that same test. Again, these are averages – we’re sure some would agree or disagree with these figures.

In test automation, if your test cases are not running faster, then you are doing something wrong. Automation should allow you to execute your test cases at least 3 to 5 times faster than conventional manual testing or 2 minutes for each test in our assumptions. LogiGear also feels that our keyword test design approach, Action Based Testing, should allow testers to write and develop tests faster; this improvement in time should be at least 2 times faster, or 3 minutes for each test in our calculations.

Lastly, we factor in the cost of developing and running these test cases both manually and automatically. The rate we use to calculate is our costs to do this work offshore in our Vietnam Testing Facility, which provides low cost high expertise testing services. The rate we use as a comparison to reflect the savings, is doing this same work in the United States.

Here is a simple matrix of our data we used for our calculation:


An Example of our Approach:

Let’s use a test example of checking a website registration system. This system has a registration dialogue and we want to test it for different inputs. Your organization might write test cases in a classic test case narrative format, like the example below:

The above method is not very friendly to automation, nor is it very efficient test case design.

However, for keyword automation, not only is the test case designed for automation, but it’s also a very efficient way for documenting test.

Below, we took the same set of tests that we created above using a traditional narrative approach with steps and expected outcomes, and reflected them as keywords.

At LogiGear, our tests look like this (plus!! we then automate these keywords):

This is a very efficient way of designing tests and they are in format that allows for easy test automation.


We’re here to help you get the best return on your testing investment possible. You can find case studies on our client success with Action Based Testing and TestArchitect.

Contact us today to find out how LogiGear can save your company money.


Here are some Case Studies that have adopted LogiGear’s low cost test services and automation methods:

{faqslider tabs}TestingROI_Casestudy{/faqslider}

LogiGear Corporation
LogiGear Corporation provides global solutions for software testing, and offers public and corporate software testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast, cost-effective results. Since 1994, LogiGear has worked with Fortune 500 companies to early-stage start-ups in, creating unique solutions to meet their clients’ needs. With facilities in the US and Viet Nam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

The Related Post

When configured with a Python harness, TestArchitect can be used to automate testing on software for custom hardware Unlike other proprietary and open source tools, that are able to automate only desktop, or mobile, TestArchitect (TA Test) has the ability to test the software that runs on hardware in the following ways: 1. TA can ...
Bringing in experts can set you up for automation success. Test automation isn’t easy when your testing gets beyond a few hundred test cases. Lots of brilliant testers and large organizations have, and continue to struggle with test automation, and not for lack of effort. Everyone understands the value of test automation, but few testing ...
The guide for CUI Automated Testing strategies, including chatbot testing and voice app testing. In the Software Testing industry, trends come and go that shape the future of testing. From Automation in Agile, to the DevOps era we are now in, trends are what evolve and improve our testing processes and ideologies. Currently, many researchers ...
An automation framework is a way to organize your code in meaningful manner so that any person who is working with you can understand what each file contains. Automation frameworks differ based on how you organize your code – it can be organized based on your data, so that any person who wants to use ...
Having the right Test Automation plan helps bridge gaps and fragmentations in the complex mobile environment. Figuring out the best Test Automation plan is one of the biggest frustrations for today’s digital teams. Organizations struggle to develop cross-platform Test Automation that can fit with their Continuous Integration cadence, their regression cycles and other elements of ...
Two dominant manual testing approaches to the software testing game are scripted and exploratory testing. In the test automation space, we have other approaches. I look at three main contexts for test automation: 1. Code context – e.g. unit testing. 2. System context – e.g. protocol or message level testing. 3. Social context – e.g. ...
Identifying which tests to begin with when starting automation is key to driving testing cycle times down and coverage up. So there you are. You’ve done a little research and made the business case to upper management regarding test automation and they bit on the proposal. Surprisingly, they supported you all the way and are extremely ...
From automotive Software Testing standards, testing techniques, and process, this article is an in-depth guide for those looking to transfer their existing skills to this exciting industry. For the Software Car, autonomous driving gets most of the hype, but most overlook the fact that there is so much more to Software Testing for the automotive ...
“Happy About Global Software Test Automation: A Discussion of Software Testing for Executives” Author: Hung Q. Nguyen, Michael Hackett, and Brent K. Whitlock Publisher: Happy About (August 1, 2006) Finally, a testing book for executives!, November 17, 2006 By Scott Barber “Chief Technologist, PerfTestPlus” Happy About Global Software Test Automation: A Discussion of Software Testing ...
Every once in a while a book is put together that should be read by every person with a relationship to software development. This book is one of them. Everyone dreams of automating their software testing, but few make it a reality. This down-to-earth book contains stories of 28 teams that went for it, including ...
Cross-Browser Testing is an integral part of the Software Testing world today. When we need to test the functionality of a website or web application, we need to do so on multiple browsers for a multitude of reasons.
Regardless of your current state of tools, building an effective Continuous Integration suite of significant automated regression tests is the key to moving to a higher level of confidence in today’s development world. In the evolution timeline of software development tools, new tools have recently proliferated. We have all been sold on collaboration, transparency and ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe