What is the Automation ROI ticker?
The LogiGear Automation Return on Investment (ROI) ticker, the set of colored numbers that you see above the page, shows how much money we presumably save our customers over time by employing test automation as compared to doing those same tests manually, both at the design and execution level.
We’ve segmented this page into three sections: one is to clarify our assumptions; then, to provide our audience some definitions, and finally to offer an example of our approach. We hope this helps clarify how we view test automation ROI.
Definitions:
What is a test case and how does LogiGear define this? A test case is a self-contained experiment — a sequence of operational and verifying actions, with a specific set of data, towards the application under test — with a defined outcome.
At LogiGear we reflect test cases as a series of actions or lines of actions. This is quite different from what most testers are used to.
In our ticker, a New Test Case is a test case that is created once during the day by our test engineering teams. We capture this number in every 24 hour cycle.
In our ticker, a Modified Test Case is a test case that has already been created, using the above criteria, but has been modified to accommodate new changes in the application. This might be a new verification point or a new navigation method. We might change that test case several times during the day, and we again, update this number in every 24 hour cycle.
How do we reflect our presumed savings?
The presumed Money Saved is calculated by comparing the total number of test cases that are developed and run automatically to the same number of tests that would be run and developed manually. As noted above in our assumption section, we make some assumptions in our calculation. One being the time it takes to develop and run our keyword based automation vs. the time it takes to develop and run a manual test case. We also factor in low cost offshore services, that maximize not only the resource cost, but the implementation costs.
Here is our simple formula reflecting costs:
As noted above – in our services organization, we generally realize a five (5) fold improvement in test execution and at least a two (2) fold improvement in designing and updating new and modified tests through our keyword method.
Assumptions and Clarifications:
The ticker reflects test cases that are developed and modified daily by our offshore service teams in Vietnam. LogiGear has been helping companies with software testing and test automation for many years. Over those years, we’ve compiled the number of tests we do for our clients as a matter of pride and internal tracking. In that time-frame we have developed over 8 million tests on 3000+ projects (yes, that’s million with an M!).
On to our numbers…
I’m sure we can all agree that executing a manual test takes more time. What we tried to do is average out the time it would normally take to run through a standard manual test case (with the assumption that these tests are required to be executed multiple iterations). We also looked at the time it would take to develop a manual test case, using the standard manual test case narrative approach.
LogiGear’s assumption is that it takes roughly 10 minutes to execute and document a standard test case manually and 6 minutes to write that same test. Again, these are averages – we’re sure some would agree or disagree with these figures.
In test automation, if your test cases are not running faster, then you are doing something wrong. Automation should allow you to execute your test cases at least 3 to 5 times faster than conventional manual testing or 2 minutes for each test in our assumptions. LogiGear also feels that our keyword test design approach, Action Based Testing, should allow testers to write and develop tests faster; this improvement in time should be at least 2 times faster, or 3 minutes for each test in our calculations.
Lastly, we factor in the cost of developing and running these test cases both manually and automatically. The rate we use to calculate is our costs to do this work offshore in our Vietnam Testing Facility, which provides low cost high expertise testing services. The rate we use as a comparison to reflect the savings, is doing this same work in the United States.
Here is a simple matrix of our data we used for our calculation:
An Example of our Approach:
Let’s use a test example of checking a website registration system. This system has a registration dialogue and we want to test it for different inputs. Your organization might write test cases in a classic test case narrative format, like the example below:
The above method is not very friendly to automation, nor is it very efficient test case design.
However, for keyword automation, not only is the test case designed for automation, but it’s also a very efficient way for documenting test.
Below, we took the same set of tests that we created above using a traditional narrative approach with steps and expected outcomes, and reflected them as keywords.
At LogiGear, our tests look like this (plus!! we then automate these keywords):
This is a very efficient way of designing tests and they are in format that allows for easy test automation.
We’re here to help you get the best return on your testing investment possible. You can find case studies on our client success with Action Based Testing and TestArchitect.
Contact us today to find out how LogiGear can save your company money.
Here are some Case Studies that have adopted LogiGear’s low cost test services and automation methods:
| {faqslider tabs}TestingROI_Casestudy{/faqslider} |
