May 2006 - Introduction to Software Testing

Rob Pirozzi, LogiGear Corporation

This article will provide a brief introduction and overview of software testing, discuss both manual and automated testing, and introduce the concept of blended testing. It is intended as a brief primer on the software testing process.

Software testing is both a discipline and a process. Though software testing is part of the software development process, it should not be considered part of software development. It is a separate discipline from software development. Software development is the process of coding functionality to meet defined end-user needs. Software testing is an iterative process of both validating functionality, and, even more important, attempting to break the software. The iterative process of software testing consists of:

• Designing tests
• Executing tests
• Identifying problems
• Getting problems fixed

The objective of software testing is to find problems and fix them to improve quality. Software testing typically represents 40% of a software development budget.

While Software testing tends to be considered a part of development, it is really its own discipline and should be tracked as its own project. Software testing, while working very closely with development, should be independent enough to be able to hold-up or slow product delivery if quality objectives are not met.

Types of Software Testing

Software testing consists of several subcategories, each of which is done for different purposes, and often using different techniques. Software testing categories include:

• Functionality testing to verify the proper functionality of the software, including validation of system and business requirements, validation of formulas and calculations, as well as testing of user interface functionality.
• Forced error testing, or attempting to break and fix the software during testing so that customers do not break it in production.
• Compatibility testing to ensure that software is compatible with various hardware platforms, operating systems, other software packages, and even previous releases of the same software.
• Performance testing to see how well software performs in terms of the speed of computations and responsiveness to the end-user.
• Scalability testing to ensure that the software will function well as the number of users and size of databases increase.
• Stress testing to see how the system performs under extreme conditions, such as a very large number of simultaneous users.
• Usability testing to ensure that the software is easy and intuitive to use.
• Application security testing to make sure that valuable and sensitive data cannot be accessed inappropriately or compromised under concerted attack.

In some cases, there may even have to be other types of testing such as regulatory-compliance testing, depending on the type of software and intended industry.

Methods of Software Testing

There are two basic methods of performing software testing:

1. Manual testing
2. Automated testing

Manual Software Testing

As the name would imply, manual software testing is the process of an individual or individuals manually testing software. This can take the form of navigating user interfaces, submitting information, or even trying to hack the software or underlying database. As one might presume, manual software testing is labor-intensive and slow. There are some things for which manual software testing is appropriate, including:

• User interface or usability testing
• Exploratory/ad hoc testing (where testers do not follow a 'script', but rather testers 'explore' the application and use their instincts to find bugs)
• Testing areas of the application which experience a lot of change.
• User acceptance testing (often, this can also be automated)

The time commitment involved with manual software testing is one of its most significant drawbacks. The time needed to fully test the system will typically range from weeks to months. Variability of results depending on who is performing the tests can also be a problem. For these reasons, many companies look to automation as a means of accelerating the software testing process while minimizing the variability of results.

Automated Software Testing

Automated software testing is the process of creating test scripts that can then be run automatically, repetitively, and through many iterations. Done properly, automated software testing can help to minimize the variability of results, speed up the testing process, increase test coverage (the number of different things tested), and ultimately provide greater confidence in the quality of the software being tested.

There are, however, some things for which automated software testing is not appropriate. These include:

• End user usability testing is not typically a good candidate for automated testing.
• Tests which will not be run more than a couple of times are typically not a good candidate for automated tasting, since the payoff of in test automation comes after many test executions.
• Tests for areas of the application which experience a lot of change are also not a good candidate for automation since this can lead to substantial maintenance of test automation scripts. Such areas of the application may be more effectively tested manually.

It is important to note that test automation is software, and just like the software you are building for internal or external customers, it must be well-architected. A good test automation architecture, such as a keyword-driven testing framework, will reduce the overall cost of ownership of your test automation by minimizing maintenance expense and increasing the number of automated tests, allowing your organization to run more tests (and achieve higher quality) for the same investment of time and money.

Achieving the Right Blend of Software Testing

An effective software testing process is typically a mix of test types, executed through a combination of manual and automated testing. The mix and number of tests is determined by the quality requirements of the application (i.e. Is it mission critical? Is time-to-market the most important factor?). Each method (automated or manual) is used for what is appropriate.

Manual testing is best leveraged for those tests which require spontaneity and creativity, as well a good deal of subjectivity:

• User interface or usability testing
• Exploratory/ad hoc testing

While automated testing is best used for tests which are explicit and repetitive:

• General QA and functionality tests (i.e. does each module do what the requirements say it should? How does the application respond to incorrect inputs?)
• 'End to end' scenario tests (simulating a 'real world' use of the software in a production environment)
• Performance, load, and stress testing

In practice, 80 to 95% of tests should be able to be automated. It is typically not the tests which limit the amount of automation, but rather a limit in the number or ability of testing staff, or technical limitations in the test automation tool to interact with the application. The right combination of tests, technology, and people can lead to a very high level of test automation and all the inherent benefits.

Conclusion

As one can see, software testing is a very large undertaking. It is very important to have an effective process in place prior to embarking on any major software development effort. This is essential to the success of the overall effort, and delivering quality software to end-users.