|
How to Measure Testing Success
By Rob Pirozzi
Introduction
All to often, senior management judges software testing though the lens of potential cost savings. Test automation and outsourcing are simplistically looked at as simply methods to reduce the costs of software testing. But the sad truth is that simply automating or offshoring a poor software testing effort will simply yield an automated and offshore poor testing effort. And if the same simplistic approach was taken in the planning of automation and offshoring, the costs may not even be lower.
It is critically important for management to define "success" for their software testing efforts. While cost is one metric, it is not the sole metric. Other issues that should also be taken into account include:
- Breadth of testing coverage
- Test reusability
- Value of the data reported by software testing
- The quality of the software under test
Using these criteria, management should come up with a clear definition of success that is consistent with their testing strategy and testing methodology. Of course, if there is no strategy and methodology, then that must be developed first - but that is a topic for another article
This paper will discuss the importance of these other non-cost success criteria.
Testing Coverage
Testing coverage refers to how much of the software actually gets tested before the software is released. While it is not possible to test absolutely everything in a large and complex software application, it is certainly desirable to test as much as possible. A reasonable place to start is to thoroughly test the most critical and most used parts of the software. It may turn out that test automation and/or offshoring may allow a company to test their software much more thoroughly in the same amount of time. In this scenario, testing costs may not actually go down, but many fewer defects will make it through to the customer which will yield down-stream cost savings in avoided in-market support and maintenance costs.
Test Reusability
A software testing effort where all tests need to be recreated with each revision of the software or release of related software is one that will be expensive regardless of automation and offshoring. A consideration for any software testing effort should be how much return are you getting on your software testing effort - how many of the tests are a reusable asset that helps to keep down ongoing test maintenance costs (see: Tests Built-to-Last: Software Tests as an Asset, by Hung Q. Nguyen).
As Test Reusability Goes Up, Test Maintenance Costs Go Down
A high degree of test reusability is an important goal for any software testing effort. Not only does it help to keep costs down, but it also allows future testing to be performed more quickly, and codifies organizational knowledge about the software under test. All are extremely beneficial and desirable outcomes.
Value of Data
Software testing can also perform an invaluable service to an organization. It can and should report clear, concise, and easy to understand metrics to engineering and senior management about the software under test (see: Understanding Metrics in Software Testing). Metrics such as test cases run, testing coverage, bugs found, and bugs fixed provide management with insight into the quality of the software under test and its readiness for market. Only with such information can management make informed ship/no-ship decisions.
It should be a goal of any software testing effort to provide management with the data that it needs to be able to make critical business decisions about the readiness for market of the software under test. Such data may help organizations to avoid shipping a product prematurely resulting in higher downstream support costs.
Software Quality
Good software testing will not in and of itself result in quality software. There really are four pillars upon which software quality rests, of which software testing is one.
The Four Pillars of Software Quality
Assuming that the processes are in place to yield quality product definition, design, and coding, then software testing can contribute greatly to the in-market quality of the software. A good, sound, software testing effort that is focused upon the right things is the difference between releasing good software with defects and quality software that is defect free.
A quality experience with the software on the part of the customer yields good-will, customer loyalty, new customers, increased revenue, and decreased support costs. All of these things can be missed when focused simply on the cost of software testing.
Conclusion
Many of these success criteria may not actually reduce the cost of software testing, but rather will provide benefits "downstream". They help an organization to avoid future costs, as well as help to accrue other benefits such as good-will, customer loyalty, new customers, and increased revenue. Overall, considering these other criteria may actually help to improve revenue, reduce costs overall, and improve profitability - even if the organization does not save a dime on software testing. These are outcomes that would be totally missed by a company solely focused on decreasing testing costs. Defining success using all metrics can yield a far superior software testing program.
Related Articles
Download software testing white papers
|
