Understanding Quality Cost

This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives.

Quality cost is the sum of all costs a company invests into the release of a quality product. When developing a software product, there are 4 types of quality costs: prevention costs, appraisal costs, internal failure costs, and external failure costs.

  1. Prevention costs represent everything a company spends to prevent software errors, documentation errors, and other product-related errors. These include requirements and usability analysis, for example. Dollars spent on prevention costs are the most effective quality dollars, because preventing errors from getting into the product is much cheaper than fixing errors later. If there is an error in a requirement or the intended usability, and money is spent on developing the software to the erroneous requirement, the costs of identifying the error, determining how to fix it, and then developing new code to correct it will arise later.
  2. Appraisal costs include the money spent on the actual testing activity. Any and all activities associated with searching for errors in the software and associated product materials falls into this category. This includes all testing by the developers themselves, by an internal test team, and by an outsourced software test organization. This also includes all associated hardware, software, labor, and other costs. Once a product is in the coding phases, the goal is to do the most effective appraisal job, so that internal failure work is streamlined, well-managed, and prevents skyrocketing external failure costs.
  3. Internal failure costs are the costs of coping with errors discovered during development and testing. These are bugs found before the product is released. As we mentioned previously, the further in the development process the errors are discovered, the more costly they are to fix. So, the later the errors are discovered, the higher their associated internal failure costs will be.
  4. External failure costs are the costs of coping with errors discovered after the product is released. These are typically errors found by your customers. These costs can be much higher than internal failure costs, because the stakes are much higher. These costs include post-release customer and technical support. Errors at this stage can also be costly in terms of your company’s reputation and may lead to lost customers.

The following table shows examples of costs that fall into each of the four categories of quality costs.

Prevention Appraisal
Staff training
Requirements analysis
Early prototyping
Fault tolerant design
Defensive programming
Usability analysis
Clear specification
Accurate internal documentation
Pre-purchase evaluation of the
reliability of development tools
Design reviews
Code inspection
Glass box testing
Black box testing
Beta testing
Test Automation
Usability testing
Pre-release out-of-box testing
by customer service staff
Internal Failure External Failure
Bug fixes
Regression testing
Wasted in-house user time
Wasted tester time
Wasted writer time
Wasted marketer time
Wasted advertisements
Direct cost of late shipment
Opportunity cost of late shipment
Technical support calls
Answer books (for support)
Investigating complaints
Refunds and recalls
Interim bug fix releases
Shipping product updates
Warranty, liability costs
PR to soften bad reviews
Lost sales
Lost customer goodwill
Supporting multiple versions in the
field
Reseller discounts to keep them
selling the product

The Four Categories of Quality Costs

All the costs mentioned above can be effectively reduced through smarter test efforts that include a high degree of Test Automation. Test Automation when done right leads to greater test coverage, resulting in higher-quality products. Higher-quality products require less technical support, fewer patches, and lead to greater customer satisfaction. Smarter Automated Testing also speeds up the release process and incrementally reduces the manual test costs. But most of all, more test coverage gives you and your customers more confidence in your product. You will feel more comfortable knowing that there are not bugs lurking in your software that have not been exposed yet because of insufficient test coverage. You will also not have to scramble at the last minute (typically, on the first day of that much-needed vacation) to deal with a problem and fix it to your customer’s satisfaction in a rush.

The solution to quality cost problems is to get a better understanding of your investment in product quality and manage your costs better. The first place most organizations look for a better understanding is in the highest cost area: the software test effort, or lack thereof. For example, if you do not test at all, your testing or appraisal cost is low. You will ship on time, but your external failure costs will skyrocket. Your prevention and appraisal costs will result in finding errors that can be corrected while they are still internal failures, where they are cheaper to deal with.

The goal of understanding quality costs is to analyze where you spend your time and money to get the most bang for the buck. It is well known that it is faster and cheaper to find and fix a bug during unit testing done by developers early in the development cycle. Should we then spend most of our time/budget on unit testing? No. There are many limitations to unit testing. Unit testing is not capable of finding many varieties of bugs, including graphical user interface (GUI) bugs, usability problems, End-to-End bugs, and configuration bugs. For most organizations, getting a better unit test effort will help you release a better product sooner. It is not a replacement for the test effort done by skilled software testers, but it may reduce the time that test effort takes. Understanding quality costs will hopefully help you shift some of your test effort to the most cost-effective places.

In the following, the total quality cost is shown in the upper bathtub-shaped curve. On the bottom axis is the quality of performance, ranging from totally defective to zero defects. On the left axis is the cost per good unit of product. You can see that with highly defective software, your prevention and appraisal costs are very low, but your failure costs are very high, yielding a high total quality cost. With zero defect software, likewise, your failure costs are very low, but your prevention and appraisal costs are very high. To optimize your total quality costs, you want to be between these extremes, at the bottom of the bathtub curve.

Theoretical Model of Optimum Quality Cost

This offers 2 challenges. First, a sufficiently sophisticated accounting system allowing a typical mid-sized company to track the total cost of quality has yet to be developed. To optimize total quality cost, you need to have the appropriate categories in your accounting system and keep track of the related costs. Second, you need to be able to track your external quality costs. You may not even have enough information from customers on why the software is not working for them. How are you going to know what to book into your accounting system for external failure costs? The point here is that while capturing this data is difficult and expensive, you know that the benefit is reducing your overall cost of quality. You need to determine if the benefits of tracking your total quality cost will give you enough of a return on investment to make setting up the appropriate accounting system and paying for the implementation of the program worthwhile.

Conclusion

Taking the time to better understand you quality costs will help you to much more cost effectively deliver a quality software product. You will be able to optimize the various costs to achieve the best quality achievable at a more reasonable price.

[1] J. M. Juran, Quality Control Handbook, 3rd Edition. New York: McGraw Hill, 1974.

Hung Nguyen

Hung Nguyen co-founded LogiGear in 1994, and is responsible for the company’s strategic direction and executive business management. His passion and relentless focus on execution and results has been the driver for the company’s innovative approach to software testing, test automation, testing tool solutions and testing education programs.

Hung is co-author of the top-selling book in the software testing field, “Testing Computer Software,” (Wiley, 2nd ed. 1993) and other publications including, “Testing Applications on the Web,” (Wiley, 1st ed. 2001, 2nd ed. 2003), and “Global Software Test Automation,” (HappyAbout Publishing, 2006). His experience prior to LogiGear includes leadership roles in software development, quality, product and business management at Spinnaker, PowerUp, Electronic Arts and Palm Computing.

Hung holds a Bachelor of Science in Quality Assurance from Cogswell Polytechnical College, and completed a Stanford Graduate School of Business Executive Program.

Rob Pirozzi

Over 20 years of sales, marketing, management, and technology experience in high technology with exposure to industries including financial services, healthcare, higher education, government, and manufacturing; demonstrating a strong track record of success. Proven ability to build and maintain strong relationships, contribute to target organization success, and deliver results. Website: http://www.robpirozzi.com/

Hung Q. Nguyen
Hung Nguyen co-founded LogiGear in 1994, and is responsible for the company’s strategic direction and executive business management. His passion and relentless focus on execution and results has been the driver for the company’s innovative approach to software testing, test automation, testing tool solutions and testing education programs. Hung is co-author of the top-selling book in the software testing field, “Testing Computer Software,” (Wiley, 2nd ed. 1993) and other publications including, “Testing Applications on the Web,” (Wiley, 1st ed. 2001, 2nd ed. 2003), and “Global Software Test Automation,” (HappyAbout Publishing, 2006). His experience prior to LogiGear includes leadership roles in software development, quality, product and business management at Spinnaker, PowerUp, Electronic Arts and Palm Computing. Hung holds a Bachelor of Science in Quality Assurance from Cogswell Polytechnical College, and completed a Stanford Graduate School of Business Executive Program.
Hung Q. Nguyen on Linkedin

The Related Post

This article was adapted from a presentation titled “How to Turn Your Testing Team Into a High-Performance Organization” to be presented by Michael Hackett, LogiGear Vice President, Business Strategy and Operations, at the Software Test & Performance Conference 2006 at the Hyatt Regency Cambridge, Massachusetts (November 7 – 9, 2006). Introduction Testing is often looked ...
The key factors for success when executing your vision.   There is an often cited quote: “…unless an organization sees that its task is to lead change, that organization—whether a business, a university, or a hospital—will not survive. In a period of rapid structural change the only organizations that survive are the ‘change leaders.’” —Peter ...
Back from more training, I was up at a client in Bellevue and really enjoyed teaching a performance class to a world class testing organization. I found that the students were very receptive to many of the concepts and ideas that the class offers.
Test design is the single biggest contributor to success in software testing. Not only can good test design result in good coverage, it is also a major contributor to efficiency. The principle of test design should be “lean and mean.” The tests should be of a manageable size and at the same time complete and ...
The Testing Domain Workbook is the most extensive and exhaustive work you will ever find on a specific testing technique (or related techniques if you include equivalence class analysis and boundary testing as the book does). What I like best is the combination of academic background and roots combined with practical experience and industrial practice. All the concepts are ...
Introduction Keyword-driven testing is a software testing technique that separates much of the programming work of test automation from the actual test design. This allows tests to be developed earlier and makes the tests easier to maintain. Some key concepts in keyword driven testing include:
As I write this article I am sitting at a table at StarEast, one of the major testing conferences. As you can expect from a testing conference, a lot of talk and discussion is about bugs and how to find them. What I have noticed in some of these discussions, however, is a lack of ...
Plan your Test Cases with these Seven Simple Steps What is a mind map? A mind map is a diagram used to visually organize information. It can be called a visual thinking tool. A mind map allows complex information to be presented in a simplified visual format. A mind map is created around a single ...
Training has to be fun. Simple as that. To inspire changed behaviors and adoption of new practices, training has to be interesting, motivating, stimulating and challenging. Training also has to be engaging enough to maintain interest, as trainers today are forced to compete with handheld mobile devices, interruptions from texting, email distractions, and people who think they ...
Please note: This article was adapted from a blog posting in Karen N. Johnson’s blog on July 24, 2007. Introduction The password field is one data entry field that needs special attention when testing an application. The password field can be important (since accessing someone’s account can start a security leak), testers should spend more ...
Dr. Cem Kaner – Director, Center for Software Testing Education & Research, Florida Institute of Technology PC World Vietnam: What did you think of VISTACON 2010? Dr. Kaner: I am very impressed that the event was very professionally organized and happy to meet my old colleagues to share and exchange more about our area of ...
Experience-based recommendations to test the brains that drive the devices In essentially every embedded system there is some sort of product testing. Typically there is a list of product-level requirements (what the product does), and a set of tests designed to make sure the product works correctly. For many products there is also a set ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe