July 2007 - The Early Evolution of Software Testing

By Hung Q. Nguyen and Rob Pirozzi

Introduction

It is very common for business processes to evolve over time. Certainly software development has undergone substantial evolution over the years. Similarly, software testing has also undergone an evolutionary process.

This article, the first of two, outlines the early phases of software testing. Software testing has already undergone two clearly identifiable phases (Software Testing 1.0 and 2.0). Both of these phases have largely under performed expectations. This paper will discuss the reasons for the shortcomings of these early phases of software testing. These early phases, however, have brought us to the threshold of the next major evolutionary development in software testing - Software Testing 3.0.

The second article in this series will discuss Software Testing 3.0 and how it delivers on the promise of software testing. It also discusses why senior management will find this evolutionary step important - in essence, what it means to them.

Software Testing 1.0

In "Software Testing 1.0", software testing was misunderstood. It was considered "uncool" if it was considered at all. In the 1.0 phase, software testing was an afterthought or adjunct to the development process. It was left to less skilled, lower-paid individuals who actually viewed software testing as the entry point on a software development career path. During the Software Testing 1.0 phase there were few useful testing tools and methods for achieving a high degree of automation. The tools that existed were expensive, complex, difficult to use, and ineffective at addressing productivity needs and concerns. Importantly, in this phase, executive management was mostly disconnected from software testing - assuming that somehow it got done.

Software Testing 2.0

"Software Testing 2.0" improved on the first phase. With "Software Testing 2.0" came the recognition that software testing was and is an important part of the development process. Software testing was now cool and everybody started to test in earnest. With the high degree of acceptance of software testing, came ownership issues - issues of where testing fits organizationally, how it is budgeted for, and from where testing should receive its direction. There was also an explosion of available tools. These tools were often the source of additional problems. Testing tools often further distracted the testing efforts, turning the testing process into one of tool selection without clear goals, architecture or direction. Within this phase, executive management's understanding and involvement with testing was still rudimentary.

The Shortcomings of Early Software Testing

Software Testing 1.0 and 2.0 both under performed from the standpoint of organizations' expectations for testing. There are some significant common threads throughout these first two phases that are at the root of unmet expectations:

1. Lack of involvement and understanding on the part of senior management - without a clear understanding of the importance of testing, its place within the development process and its valuable contributions to the organization, testing can receive misguided direction, inadequate funding, and be subject to unrealistic expectations.
2. Lack of guiding strategy - In the early phases, especially Software Testing 2.0, the emphasis was on testing for testing's sake, and tools and tool selection without any strategy for guidance. Testing and tool selection without a strategy will provide an organization with a software testing program, but the chances that it will provide better quality products through better testing coverage, in less time and at less expense, are very slim.

In essence, the first two phases yielded testing programs that were ill-equipped to actually meet the needs of the organization for better software made faster and cheaper.

These shortcomings yield complaints that are all too common for 1.0/2.0 software testing programs, such as:

• "Our organization struggles with test automation and off-shoring and we do not see significant gains in coverage or quality, or reduced costs."
• "Software testing requires too much hands-on."
• "Software quality problems are as pervasive as ever in spite of advancements in development techniques and technologies."

Testing is emerging into a healthy industry and an established discipline. According to IDC, software testing is a $13 billion business worldwide (Worldwide and U.S. Offshore IT Services 2006-2010, 2006). Yet, although executive management has a much clearer vision of the marketing, sales and finance parts of the product release process, most executives still lack an understanding of the challenges of quality and software testing, or they understand the challenges but have little confidence in the proposed solutions. This is exacerbated by having little or no visibility into the effectiveness - or rather, the ineffectiveness - of testing. As a result, off-shoring of software testing has been unsatisfactory due to problematic communications, insufficient or mismatched skill sets at the outsourced software test organization, and problems with management, vendor relations and global infrastructure. Furthermore, test automation implementation has also been failing due to many pitfalls, including uncertain return-on-investment, lack of visibility, and poor reusability, scalability and maintainability.

Conclusion

Software Testing 1.0 and 2.0 have failed to meet expectations because of lack of understanding and involvement on the part of senior management, as well as the lack of a guiding strategy. Management clearly needs to have an understanding of the challenges of software testing and greater confidence in proposed solutions. They need visibility into testing and its effectiveness or lack thereof. In addition, an organization needs to have an effective overall strategy for software testing to act as a road-map - a strategy that can help to guide them with process improvement, tool selection, and outsourcing efforts. These are the challenges that are met by the next evolutionary phase of software testing, Software Testing 3.0, the subject of the next article in this series.