home Test Methods & Metrics How to Measure Testing Success

How to Measure Testing Success

Introduction

All to often, senior management judges software testing though the lens of potential cost savings. Test automation and outsourcing are simplistically looked at as simply methods to reduce the costs of software testing. But the sad truth is that simply automating or offshoring a poor software testing effort will simply yield an automated and offshore poor testing effort. And if the same simplistic approach was taken in the planning of automation and offshoring, the costs may not even be lower.

It is critically important for management to define “success” for their software testing efforts. While cost is one metric, it is not the sole metric. Other issues that should also be taken into account include:

  • Breadth of testing coverage
  • Test reusability
  • Value of the data reported by software testing
  • The quality of the software under test

Using these criteria, management should come up with a clear definition of success that is consistent with their testing strategy and testing methodology. Of course, if there is no strategy and methodology, then that must be developed first – but that is a topic for another article

This paper will discuss the importance of these other non-cost success criteria.

Testing Coverage

Testing coverage refers to how much of the software actually gets tested before the software is released. While it is not possible to test absolutely everything in a large and complex software application, it is certainly desirable to test as much as possible. A reasonable place to start is to thoroughly test the most critical and most used parts of the software. It may turn out that test automation and/or offshoring may allow a company to test their software much more thoroughly in the same amount of time. In this scenario, testing costs may not actually go down, but many fewer defects will make it through to the customer which will yield down-stream cost savings in avoided in-market support and maintenance costs.

Test Reusability

A software testing effort where all tests need to be recreated with each revision of the software or release of related software is one that will be expensive regardless of automation and offshoring. A consideration for any software testing effort should be how much return are you getting on your software testing effort – how many of the tests are a reusable asset that helps to keep down ongoing test maintenance costs (see: Tests Built-to-Last: Software Tests as an Asset, by Hung Q. Nguyen).

As Test Reusability Goes Up, Test Maintenance Costs Go Down

A high degree of test reusability is an important goal for any software testing effort. Not only does it help to keep costs down, but it also allows future testing to be performed more quickly, and codifies organizational knowledge about the software under test. All are extremely beneficial and desirable outcomes.

Value of Data

Software testing can also perform an invaluable service to an organization. It can and should report clear, concise, and easy to understand metrics to engineering and senior management about the software under test (see: Understanding Metrics in Software Testing). Metrics such as test cases run, testing coverage, bugs found, and bugs fixed provide management with insight into the quality of the software under test and its readiness for market. Only with such information can management make informed ship/no-ship decisions.

It should be a goal of any software testing effort to provide management with the data that it needs to be able to make critical business decisions about the readiness for market of the software under test. Such data may help organizations to avoid shipping a product prematurely resulting in higher downstream support costs.

Software Quality

Good software testing will not in and of itself result in quality software. There really are four pillars upon which software quality rests, of which software testing is one.

The Four Pillars of Software Quality

Assuming that the processes are in place to yield quality product definition, design, and coding, then software testing can contribute greatly to the in-market quality of the software. A good, sound, software testing effort that is focused upon the right things is the difference between releasing good software with defects and quality software that is defect free.

A quality experience with the software on the part of the customer yields good-will, customer loyalty, new customers, increased revenue, and decreased support costs. All of these things can be missed when focused simply on the cost of software testing.

Conclusion

Many of these success criteria may not actually reduce the cost of software testing, but rather will provide benefits “downstream”. They help an organization to avoid future costs, as well as help to accrue other benefits such as good-will, customer loyalty, new customers, and increased revenue. Overall, considering these other criteria may actually help to improve revenue, reduce costs overall, and improve profitability – even if the organization does not save a dime on software testing. These are outcomes that would be totally missed by a company solely focused on decreasing testing costs. Defining success using all metrics can yield a far superior software testing program.

Hung Nguyen

Hung Nguyen co-founded LogiGear in 1994, and is responsible for the company’s strategic direction and executive business management. His passion and relentless focus on execution and results has been the driver for the company’s innovative approach to software testing, test automation, testing tool solutions and testing education programs.

Hung is co-author of the top-selling book in the software testing field, “Testing Computer Software,” (Wiley, 2nd ed. 1993) and other publications including, “Testing Applications on the Web,” (Wiley, 1st ed. 2001, 2nd ed. 2003), and “Global Software Test Automation,” (HappyAbout Publishing, 2006). His experience prior to LogiGear includes leadership roles in software development, quality, product and business management at Spinnaker, PowerUp, Electronic Arts and Palm Computing.

Hung holds a Bachelor of Science in Quality Assurance from Cogswell Polytechnical College, and completed a Stanford Graduate School of Business Executive Program.

Facebooktwittergoogle_plusredditpinterestlinkedinmail
Hung Q. Nguyen
Hung Nguyen co-founded LogiGear in 1994, and is responsible for the company’s strategic direction and executive business management. His passion and relentless focus on execution and results has been the driver for the company’s innovative approach to software testing, test automation, testing tool solutions and testing education programs.
Hung is co-author of the top-selling book in the software testing field, “Testing Computer Software,” (Wiley, 2nd ed. 1993) and other publications including, “Testing Applications on the Web,” (Wiley, 1st ed. 2001, 2nd ed. 2003), and “Global Software Test Automation,” (HappyAbout Publishing, 2006). His experience prior to LogiGear includes leadership roles in software development, quality, product and business management at Spinnaker, PowerUp, Electronic Arts and Palm Computing.
Hung holds a Bachelor of Science in Quality Assurance from Cogswell Polytechnical College, and completed a Stanford Graduate School of Business Executive Program.
Hung Q. Nguyen on Linkedin

One thought on “How to Measure Testing Success

  1. This is a good article & good site.Thank you for sharing this article. It is help us following categorize:
    healthcare, e commerce, programming, it consulting, retail, manufacturing, CRM, digital supply chain management, Delivering high-quality service for your business applications,
    Solutions for all Industries,
    Getting your applications talking is the key to better business processes,
    Rapid web services solutions for real business problems,
    Web-based Corporate Document Management System,
    Outsourcing Solution,
    Financial and Operations Business Intelligence Solution,

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe