Understanding Quality Cost

This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives.

Quality cost is the sum of all costs a company invests into the release of a quality product. When developing a software product, there are 4 types of quality costs: prevention costs, appraisal costs, internal failure costs, and external failure costs.

  1. Prevention costs represent everything a company spends to prevent software errors, documentation errors, and other product-related errors. These include requirements and usability analysis, for example. Dollars spent on prevention costs are the most effective quality dollars, because preventing errors from getting into the product is much cheaper than fixing errors later. If there is an error in a requirement or the intended usability, and money is spent on developing the software to the erroneous requirement, the costs of identifying the error, determining how to fix it, and then developing new code to correct it will arise later.
  2. Appraisal costs include the money spent on the actual testing activity. Any and all activities associated with searching for errors in the software and associated product materials falls into this category. This includes all testing by the developers themselves, by an internal test team, and by an outsourced software test organization. This also includes all associated hardware, software, labor, and other costs. Once a product is in the coding phases, the goal is to do the most effective appraisal job, so that internal failure work is streamlined, well-managed, and prevents skyrocketing external failure costs.
  3. Internal failure costs are the costs of coping with errors discovered during development and testing. These are bugs found before the product is released. As we mentioned previously, the further in the development process the errors are discovered, the more costly they are to fix. So, the later the errors are discovered, the higher their associated internal failure costs will be.
  4. External failure costs are the costs of coping with errors discovered after the product is released. These are typically errors found by your customers. These costs can be much higher than internal failure costs, because the stakes are much higher. These costs include post-release customer and technical support. Errors at this stage can also be costly in terms of your company’s reputation and may lead to lost customers.

The following table shows examples of costs that fall into each of the four categories of quality costs.

Prevention Appraisal
Staff training
Requirements analysis
Early prototyping
Fault tolerant design
Defensive programming
Usability analysis
Clear specification
Accurate internal documentation
Pre-purchase evaluation of the
reliability of development tools
Design reviews
Code inspection
Glass box testing
Black box testing
Beta testing
Test Automation
Usability testing
Pre-release out-of-box testing
by customer service staff
Internal Failure External Failure
Bug fixes
Regression testing
Wasted in-house user time
Wasted tester time
Wasted writer time
Wasted marketer time
Wasted advertisements
Direct cost of late shipment
Opportunity cost of late shipment
Technical support calls
Answer books (for support)
Investigating complaints
Refunds and recalls
Interim bug fix releases
Shipping product updates
Warranty, liability costs
PR to soften bad reviews
Lost sales
Lost customer goodwill
Supporting multiple versions in the
field
Reseller discounts to keep them
selling the product

The Four Categories of Quality Costs

All the costs mentioned above can be effectively reduced through smarter test efforts that include a high degree of Test Automation. Test Automation when done right leads to greater test coverage, resulting in higher-quality products. Higher-quality products require less technical support, fewer patches, and lead to greater customer satisfaction. Smarter Automated Testing also speeds up the release process and incrementally reduces the manual test costs. But most of all, more test coverage gives you and your customers more confidence in your product. You will feel more comfortable knowing that there are not bugs lurking in your software that have not been exposed yet because of insufficient test coverage. You will also not have to scramble at the last minute (typically, on the first day of that much-needed vacation) to deal with a problem and fix it to your customer’s satisfaction in a rush.

The solution to quality cost problems is to get a better understanding of your investment in product quality and manage your costs better. The first place most organizations look for a better understanding is in the highest cost area: the software test effort, or lack thereof. For example, if you do not test at all, your testing or appraisal cost is low. You will ship on time, but your external failure costs will skyrocket. Your prevention and appraisal costs will result in finding errors that can be corrected while they are still internal failures, where they are cheaper to deal with.

The goal of understanding quality costs is to analyze where you spend your time and money to get the most bang for the buck. It is well known that it is faster and cheaper to find and fix a bug during unit testing done by developers early in the development cycle. Should we then spend most of our time/budget on unit testing? No. There are many limitations to unit testing. Unit testing is not capable of finding many varieties of bugs, including graphical user interface (GUI) bugs, usability problems, End-to-End bugs, and configuration bugs. For most organizations, getting a better unit test effort will help you release a better product sooner. It is not a replacement for the test effort done by skilled software testers, but it may reduce the time that test effort takes. Understanding quality costs will hopefully help you shift some of your test effort to the most cost-effective places.

In the following, the total quality cost is shown in the upper bathtub-shaped curve. On the bottom axis is the quality of performance, ranging from totally defective to zero defects. On the left axis is the cost per good unit of product. You can see that with highly defective software, your prevention and appraisal costs are very low, but your failure costs are very high, yielding a high total quality cost. With zero defect software, likewise, your failure costs are very low, but your prevention and appraisal costs are very high. To optimize your total quality costs, you want to be between these extremes, at the bottom of the bathtub curve.

Theoretical Model of Optimum Quality Cost

This offers 2 challenges. First, a sufficiently sophisticated accounting system allowing a typical mid-sized company to track the total cost of quality has yet to be developed. To optimize total quality cost, you need to have the appropriate categories in your accounting system and keep track of the related costs. Second, you need to be able to track your external quality costs. You may not even have enough information from customers on why the software is not working for them. How are you going to know what to book into your accounting system for external failure costs? The point here is that while capturing this data is difficult and expensive, you know that the benefit is reducing your overall cost of quality. You need to determine if the benefits of tracking your total quality cost will give you enough of a return on investment to make setting up the appropriate accounting system and paying for the implementation of the program worthwhile.

Conclusion

Taking the time to better understand you quality costs will help you to much more cost effectively deliver a quality software product. You will be able to optimize the various costs to achieve the best quality achievable at a more reasonable price.

[1] J. M. Juran, Quality Control Handbook, 3rd Edition. New York: McGraw Hill, 1974.

Hung Nguyen

Hung Nguyen co-founded LogiGear in 1994, and is responsible for the company’s strategic direction and executive business management. His passion and relentless focus on execution and results has been the driver for the company’s innovative approach to software testing, test automation, testing tool solutions and testing education programs.

Hung is co-author of the top-selling book in the software testing field, “Testing Computer Software,” (Wiley, 2nd ed. 1993) and other publications including, “Testing Applications on the Web,” (Wiley, 1st ed. 2001, 2nd ed. 2003), and “Global Software Test Automation,” (HappyAbout Publishing, 2006). His experience prior to LogiGear includes leadership roles in software development, quality, product and business management at Spinnaker, PowerUp, Electronic Arts and Palm Computing.

Hung holds a Bachelor of Science in Quality Assurance from Cogswell Polytechnical College, and completed a Stanford Graduate School of Business Executive Program.

Rob Pirozzi

Over 20 years of sales, marketing, management, and technology experience in high technology with exposure to industries including financial services, healthcare, higher education, government, and manufacturing; demonstrating a strong track record of success. Proven ability to build and maintain strong relationships, contribute to target organization success, and deliver results. Website: http://www.robpirozzi.com/

Hung Q. Nguyen
Hung Nguyen co-founded LogiGear in 1994, and is responsible for the company’s strategic direction and executive business management. His passion and relentless focus on execution and results has been the driver for the company’s innovative approach to software testing, test automation, testing tool solutions and testing education programs. Hung is co-author of the top-selling book in the software testing field, “Testing Computer Software,” (Wiley, 2nd ed. 1993) and other publications including, “Testing Applications on the Web,” (Wiley, 1st ed. 2001, 2nd ed. 2003), and “Global Software Test Automation,” (HappyAbout Publishing, 2006). His experience prior to LogiGear includes leadership roles in software development, quality, product and business management at Spinnaker, PowerUp, Electronic Arts and Palm Computing. Hung holds a Bachelor of Science in Quality Assurance from Cogswell Polytechnical College, and completed a Stanford Graduate School of Business Executive Program.
Hung Q. Nguyen on Linkedin

The Related Post

The V-Model for Software Development specifies 4 kinds of testing: Unit Testing Integration Testing System Testing Acceptance Testing You can find more information here (Wikipedia): http://en.wikipedia.org/wiki/V-Model_%28software_development%29#Validation_Phases What I’m finding is that of those only the Unit Testing is clear to me. The other kinds maybe good phases in a project, but for test design it ...
  Explore It! is one of the very best software testing books ever written. It is packed with great ideas and Elisabeth Hendrickson’s writing style makes it very enjoyable to read. Hendrickson has a well-deserved reputation in the global software testing community as someone who has the enviable ability to clearly communicate highly-practical, well-thought-out ideas. ...
There are many ways to approach test design. These approaches range from checklists to very precise algorithms in which test conditions are combined to achieve the most efficiency in testing. There are situations, such as in testing mobile applications, complex systems and cyber security, where tests need to be creative, cover a lot of functionality, ...
The Testing Domain Workbook is the most extensive and exhaustive work you will ever find on a specific testing technique (or related techniques if you include equivalence class analysis and boundary testing as the book does). What I like best is the combination of academic background and roots combined with practical experience and industrial practice. All the concepts are ...
At VISTACON 2011, Jane sat down with LogiGear Sr. VP, Michael Hackett, to discuss complex systems.
Test design is the single biggest contributor to success in software testing. Not only can good test design result in good coverage, it is also a major contributor to efficiency. The principle of test design should be “lean and mean.” The tests should be of a manageable size and at the same time complete and ...
“Combinatorial testing can detect hard-to-find software faults more efficiently than manual test case selection methods.” Developers of large data-intensive software often notice an interesting—though not surprising—phenomenon: When usage of an application jumps dramatically, components that have operated for months without trouble suddenly develop previously undetected errors. For example, newly added customers may have account records ...
Introduction This 2 article series describes activities that are central to successfully integrating application performance testing into an Agile process. The activities described here specifically target performance specialists who are new to the practice of fully integrating performance testing into an Agile or other iteratively-based process, though many of the concepts and considerations can be ...
Introduction Software Testing 3.0 is a strategic end-to-end framework for change based upon a strategy to drive testing activities, tool selection, and people development that finally delivers on the promise of Software Testing. For more details on the evolution of Software Testing and Software Testing 3.0 see: The Early Evolution of Software Testing Software Testing ...
As I write this article I am sitting at a table at StarEast, one of the major testing conferences. As you can expect from a testing conference, a lot of talk and discussion is about bugs and how to find them. What I have noticed in some of these discussions, however, is a lack of ...
I’ve been reviewing a lot of test plans recently. As I review them, I’ve compiled this list of things I look for in a well written test plan document. Here’s a brain dump of things I check for, in no particular order, of course, and it is by no means a complete list. That said, if you ...
David S. Janzen – Associate Professor of Computer Science Department California Polytechnic State University, San Luis Obispo – homepage LogiGear: How did you get into software testing and what do you find interesting about it? Professor Janzen: The thing I enjoy most about computing is creating something that helps people. Since my first real job ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe