Home Test Methods & Metrics Understanding Quality Cost

Understanding Quality Cost

This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives.

Quality cost is the sum of all costs a company invests into the release of a quality product. When developing a software product, there are 4 types of quality costs: prevention costs, appraisal costs, internal failure costs, and external failure costs.

  1. Prevention costs represent everything a company spends to prevent software errors, documentation errors, and other product-related errors. These include requirements and usability analysis, for example. Dollars spent on prevention costs are the most effective quality dollars, because preventing errors from getting into the product is much cheaper than fixing errors later. If there is an error in a requirement or the intended usability, and money is spent on developing the software to the erroneous requirement, the costs of identifying the error, determining how to fix it, and then developing new code to correct it will arise later.
  2. Appraisal costs include the money spent on the actual testing activity. Any and all activities associated with searching for errors in the software and associated product materials falls into this category. This includes all testing by the developers themselves, by an internal test team, and by an outsourced software test organization. This also includes all associated hardware, software, labor, and other costs. Once a product is in the coding phases, the goal is to do the most effective appraisal job, so that internal failure work is streamlined, well-managed, and prevents skyrocketing external failure costs.
  3. Internal failure costs are the costs of coping with errors discovered during development and testing. These are bugs found before the product is released. As we mentioned previously, the further in the development process the errors are discovered, the more costly they are to fix. So, the later the errors are discovered, the higher their associated internal failure costs will be.
  4. External failure costs are the costs of coping with errors discovered after the product is released. These are typically errors found by your customers. These costs can be much higher than internal failure costs, because the stakes are much higher. These costs include post-release customer and technical support. Errors at this stage can also be costly in terms of your company’s reputation and may lead to lost customers.

The following table shows examples of costs that fall into each of the four categories of quality costs.

Prevention Appraisal
Staff training
Requirements analysis
Early prototyping
Fault tolerant design
Defensive programming
Usability analysis
Clear specification
Accurate internal documentation
Pre-purchase evaluation of the
reliability of development tools
Design reviews
Code inspection
Glass box testing
Black box testing
Beta testing
Test Automation
Usability testing
Pre-release out-of-box testing
by customer service staff
Internal Failure External Failure
Bug fixes
Regression testing
Wasted in-house user time
Wasted tester time
Wasted writer time
Wasted marketer time
Wasted advertisements
Direct cost of late shipment
Opportunity cost of late shipment
Technical support calls
Answer books (for support)
Investigating complaints
Refunds and recalls
Interim bug fix releases
Shipping product updates
Warranty, liability costs
PR to soften bad reviews
Lost sales
Lost customer goodwill
Supporting multiple versions in the
field
Reseller discounts to keep them
selling the product

The Four Categories of Quality Costs

All the costs mentioned above can be effectively reduced through smarter test efforts that include a high degree of Test Automation. Test Automation when done right leads to greater test coverage, resulting in higher-quality products. Higher-quality products require less technical support, fewer patches, and lead to greater customer satisfaction. Smarter Automated Testing also speeds up the release process and incrementally reduces the manual test costs. But most of all, more test coverage gives you and your customers more confidence in your product. You will feel more comfortable knowing that there are not bugs lurking in your software that have not been exposed yet because of insufficient test coverage. You will also not have to scramble at the last minute (typically, on the first day of that much-needed vacation) to deal with a problem and fix it to your customer’s satisfaction in a rush.

The solution to quality cost problems is to get a better understanding of your investment in product quality and manage your costs better. The first place most organizations look for a better understanding is in the highest cost area: the software test effort, or lack thereof. For example, if you do not test at all, your testing or appraisal cost is low. You will ship on time, but your external failure costs will skyrocket. Your prevention and appraisal costs will result in finding errors that can be corrected while they are still internal failures, where they are cheaper to deal with.

The goal of understanding quality costs is to analyze where you spend your time and money to get the most bang for the buck. It is well known that it is faster and cheaper to find and fix a bug during unit testing done by developers early in the development cycle. Should we then spend most of our time/budget on unit testing? No. There are many limitations to unit testing. Unit testing is not capable of finding many varieties of bugs, including graphical user interface (GUI) bugs, usability problems, End-to-End bugs, and configuration bugs. For most organizations, getting a better unit test effort will help you release a better product sooner. It is not a replacement for the test effort done by skilled software testers, but it may reduce the time that test effort takes. Understanding quality costs will hopefully help you shift some of your test effort to the most cost-effective places.

In the following, the total quality cost is shown in the upper bathtub-shaped curve. On the bottom axis is the quality of performance, ranging from totally defective to zero defects. On the left axis is the cost per good unit of product. You can see that with highly defective software, your prevention and appraisal costs are very low, but your failure costs are very high, yielding a high total quality cost. With zero defect software, likewise, your failure costs are very low, but your prevention and appraisal costs are very high. To optimize your total quality costs, you want to be between these extremes, at the bottom of the bathtub curve.

Theoretical Model of Optimum Quality Cost

This offers 2 challenges. First, a sufficiently sophisticated accounting system allowing a typical mid-sized company to track the total cost of quality has yet to be developed. To optimize total quality cost, you need to have the appropriate categories in your accounting system and keep track of the related costs. Second, you need to be able to track your external quality costs. You may not even have enough information from customers on why the software is not working for them. How are you going to know what to book into your accounting system for external failure costs? The point here is that while capturing this data is difficult and expensive, you know that the benefit is reducing your overall cost of quality. You need to determine if the benefits of tracking your total quality cost will give you enough of a return on investment to make setting up the appropriate accounting system and paying for the implementation of the program worthwhile.

Conclusion

Taking the time to better understand you quality costs will help you to much more cost effectively deliver a quality software product. You will be able to optimize the various costs to achieve the best quality achievable at a more reasonable price.

[1] J. M. Juran, Quality Control Handbook, 3rd Edition. New York: McGraw Hill, 1974.

Hung Nguyen

Hung Nguyen co-founded LogiGear in 1994, and is responsible for the company’s strategic direction and executive business management. His passion and relentless focus on execution and results has been the driver for the company’s innovative approach to software testing, test automation, testing tool solutions and testing education programs.

Hung is co-author of the top-selling book in the software testing field, “Testing Computer Software,” (Wiley, 2nd ed. 1993) and other publications including, “Testing Applications on the Web,” (Wiley, 1st ed. 2001, 2nd ed. 2003), and “Global Software Test Automation,” (HappyAbout Publishing, 2006). His experience prior to LogiGear includes leadership roles in software development, quality, product and business management at Spinnaker, PowerUp, Electronic Arts and Palm Computing.

Hung holds a Bachelor of Science in Quality Assurance from Cogswell Polytechnical College, and completed a Stanford Graduate School of Business Executive Program.

Rob Pirozzi

Over 20 years of sales, marketing, management, and technology experience in high technology with exposure to industries including financial services, healthcare, higher education, government, and manufacturing; demonstrating a strong track record of success. Proven ability to build and maintain strong relationships, contribute to target organization success, and deliver results. Website: http://www.robpirozzi.com/

Facebooktwitterredditpinterestlinkedinmail
Hung Q. Nguyen
Hung Nguyen co-founded LogiGear in 1994, and is responsible for the company’s strategic direction and executive business management. His passion and relentless focus on execution and results has been the driver for the company’s innovative approach to software testing, test automation, testing tool solutions and testing education programs.
Hung is co-author of the top-selling book in the software testing field, “Testing Computer Software,” (Wiley, 2nd ed. 1993) and other publications including, “Testing Applications on the Web,” (Wiley, 1st ed. 2001, 2nd ed. 2003), and “Global Software Test Automation,” (HappyAbout Publishing, 2006). His experience prior to LogiGear includes leadership roles in software development, quality, product and business management at Spinnaker, PowerUp, Electronic Arts and Palm Computing.
Hung holds a Bachelor of Science in Quality Assurance from Cogswell Polytechnical College, and completed a Stanford Graduate School of Business Executive Program.
Hung Q. Nguyen on Linkedin

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe