Automation and Requirements

Introduction

In many of the Test Automation projects that we are involved with using our Action-Based Testing methodology, management has expressed a need to relate tests and test results to system requirements. The underlying thought is that automation will create extra possibilities to control the level of compliance to requirements of the system under test. In this article I will explore some of the aspects and considerations regarding tests and requirements.

System and Test Requirements with Action-Based Testing

Action-Based Testing encourages testers to create “test requirements” as part of the test modules (spreadsheets with the tests to be carried out). The test requirements govern the scope of test cases that are in the test module; in other words, they detail the intention the tester has with the cases described.

For example, a system requirement is typically formulated like “the system should.” A test requirement typically reads as “test if.” As a short hand, we often leave the “test if” out, so a test requirement could read “the system does not accept password shorter than 6 characters,” to be understood as “test if the system does not accept password shorter than 6 characters.”

Now the next question is: how do the test requirements relate to system requirements? In many cases the link is very direct. As illustrated in the example above on minimum password length, the test requirement will relate one-on-one to a system requirement. In other cases a system requirement is a composite statement, which would lead to several test requirements.

For example “the system should enforce that passwords are at least 6 characters in length and contain at least 1 digit,” would lead to (1) “the system does not accept passwords shorter than 6 characters,” and (2) “the system requires a password to have at least 1 digit.”

Good Test Design: Going Beyond System Requirements

Another possibility is that a test requirement specifies that a test be created that does not directly relate to any system requirement. For example, a valid test requirement could state “test with 5 users from different states who are using the same mailbox.” This means the test should create a test case in which the described situation is simulated, but there is very likely no system requirement talking about 5 users.

Test requirements that are unrelated to system requirements illustrate an important consideration regarding system requirements and tests: A test designer should use system requirements as input and make sure all of them are covered in test cases, but a good test designer should go beyond that narrow mission. System requirements mostly focus on single elements of a system under test. A tester should make sure that the elements of the system work well together, and that the system can handle complex situations that the developer might not necessarily have thought about, because they were not specifically addressed in system requirements.

One potential benefit of Automation is to provide information related to system requirements. However, this needs to be carefully organized to be effective. Consider the following picture in which some relationships have been illustrated.

The solid arrows show how system requirements are, directly or indirectly, related to test requirements, which in turn are related to test cases, which in turn are related to test results. A test tool like TestArchitect will give you the possibility to establish relations 1-3 in the above picture as follows:

  1. In the details panel for a test requirement it is possible to register a “source”, this field can contain the identifier of the system requirement to which the test requirement relates.
  2. The same detail panel also shows the test cases in the test modules (after they have been created and the module has been checked in at least once), and checkmarks to indicate which one(s) cover the test requirement.
  3. The relation from the test results to the test case will be shown in the test results. Once the test result is checked in the test case details panel will show the latest result.

To easily obtain the consolidated relation (4) make sure the “source” fields of the test requirements are consistently populated, the test requirements are related to test cases and test run results (of successful runs) are checked in. This will make it easy to use the report generator in TestArchitect to show the test results per system requirement: define a report that shows test requirements, and as grouping field specify the “source” field.

Conclusion

This article has outlined that good test design has 3 major components:

  • Creating test cases that directly relate to system requirements
  • Creating test cases that indirectly relate to system requirements
  • Creating test cases that do not directly relate to system requirements

The third item is often overlooked and can be very critical in a successful Test Automation effort.

More Information

  • More information about Action-Based Testing may be found here.
  • More information on TestArchitect may be found here.
Hans Buwalda
Hans Buwalda, CTO of LogiGear, is a pioneer of the Action Based and Soap Opera methodologies of testing and automation, and lead developer of TestArchitect, LogiGear’s keyword-based toolset for software test design, automation and management. He is co-author of Integrated Test Design and Automation, and a frequent speaker at test conferences.

The Related Post

From automotive Software Testing standards, testing techniques, and process, this article is an in-depth guide for those looking to transfer their existing skills to this exciting industry. For the Software Car, autonomous driving gets most of the hype, but most overlook the fact that there is so much more to Software Testing for the automotive ...
As I wrote in various articles, organization is one of the 3 key requisites for successful automated testing, the other two being test design and automation architecture.
Looking for a solution to test your voice apps across devices and platforms? Whether you’re new or experienced in testing voice apps such as Alexa skill or Google Home actions, this article will give you a holistic view of the challenges of executing software testing for voice-based apps. It also explores some of the basic ...
The Cloud demands that we be as nimble as possible, delivering features and fixes in almost real-time fashion. Both customer and provider rely on software development that can maintain quality while being light on its feet and constantly moving. In addition, Cloud-oriented systems tend to be highly complex and dynamic in structure — more than ...
Test Automation is significant and growing-yet I have read many forum comments and blog posts about Test Automation not delivering as expected. It’s true that test automation can improve reliability while minimizing variability in the results, speed up the process, increase test coverage, and ultimately provide greater confidence in the quality of the software being ...
*You can check the answer key here
The following is a transcript of a May 7, 2008 interview with Hung Q. Nguyen, founder and CEO of LogiGear Corporation and coauthor of the best selling textbook Testing Computer Software. Interviewer: When it comes to software testing, what concerns or issues are you hearing from software developers? Hung Q. Nguyen: The most pressing concern ...
“Testing Applications on the web” – 2nd EditionAuthors: Hung Q. Nguyen, Bob Johnson, Michael HackettPublisher: Wiley; edition (May 16, 2003) This is good book. If you test web apps, you should buy it!, April 20, 2001By Dr. Cem Kaner – Director of Florida Institute of Technology’s Center for Software Testing Education & Research Book Reviews ...
Over the years, we’ve provided an extensive number of articles that provide a wealth of knowledge about Test Automation. Below is a compilation of some of those articles. Guide to Automated Voice Apps Testing This article explores some of the basic test tools you’ll need and how to blend the use different automated testing tools ...
An Overview of Four Methods for Systematic Test Design Strategy Many people test, but few people use the well-known black-box and white-box test design techniques. The technique most used, however, seems to be testing randomly chosen valid values, followed by error guessing, exploratory testing and the like. Could it be that the more systematic test ...
Introduction Many executives have some very basic questions about Software Testing. These questions address the elements of quality (customer satisfaction) and money (spending the least amount of money to prevent future loss). The basic questions that executive have about Software Testing include: Why care about and spend money on testing? Why should testing be treated ...
Bringing in experts can set you up for automation success. Test automation isn’t easy when your testing gets beyond a few hundred test cases. Lots of brilliant testers and large organizations have, and continue to struggle with test automation, and not for lack of effort. Everyone understands the value of test automation, but few testing ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe