Automation and Requirements

Introduction

In many of the Test Automation projects that we are involved with using our Action-Based Testing methodology, management has expressed a need to relate tests and test results to system requirements. The underlying thought is that automation will create extra possibilities to control the level of compliance to requirements of the system under test. In this article I will explore some of the aspects and considerations regarding tests and requirements.

System and Test Requirements with Action-Based Testing

Action-Based Testing encourages testers to create “test requirements” as part of the test modules (spreadsheets with the tests to be carried out). The test requirements govern the scope of test cases that are in the test module; in other words, they detail the intention the tester has with the cases described.

For example, a system requirement is typically formulated like “the system should.” A test requirement typically reads as “test if.” As a short hand, we often leave the “test if” out, so a test requirement could read “the system does not accept password shorter than 6 characters,” to be understood as “test if the system does not accept password shorter than 6 characters.”

Now the next question is: how do the test requirements relate to system requirements? In many cases the link is very direct. As illustrated in the example above on minimum password length, the test requirement will relate one-on-one to a system requirement. In other cases a system requirement is a composite statement, which would lead to several test requirements.

For example “the system should enforce that passwords are at least 6 characters in length and contain at least 1 digit,” would lead to (1) “the system does not accept passwords shorter than 6 characters,” and (2) “the system requires a password to have at least 1 digit.”

Good Test Design: Going Beyond System Requirements

Another possibility is that a test requirement specifies that a test be created that does not directly relate to any system requirement. For example, a valid test requirement could state “test with 5 users from different states who are using the same mailbox.” This means the test should create a test case in which the described situation is simulated, but there is very likely no system requirement talking about 5 users.

Test requirements that are unrelated to system requirements illustrate an important consideration regarding system requirements and tests: A test designer should use system requirements as input and make sure all of them are covered in test cases, but a good test designer should go beyond that narrow mission. System requirements mostly focus on single elements of a system under test. A tester should make sure that the elements of the system work well together, and that the system can handle complex situations that the developer might not necessarily have thought about, because they were not specifically addressed in system requirements.

One potential benefit of Automation is to provide information related to system requirements. However, this needs to be carefully organized to be effective. Consider the following picture in which some relationships have been illustrated.

The solid arrows show how system requirements are, directly or indirectly, related to test requirements, which in turn are related to test cases, which in turn are related to test results. A test tool like TestArchitect will give you the possibility to establish relations 1-3 in the above picture as follows:

  1. In the details panel for a test requirement it is possible to register a “source”, this field can contain the identifier of the system requirement to which the test requirement relates.
  2. The same detail panel also shows the test cases in the test modules (after they have been created and the module has been checked in at least once), and checkmarks to indicate which one(s) cover the test requirement.
  3. The relation from the test results to the test case will be shown in the test results. Once the test result is checked in the test case details panel will show the latest result.

To easily obtain the consolidated relation (4) make sure the “source” fields of the test requirements are consistently populated, the test requirements are related to test cases and test run results (of successful runs) are checked in. This will make it easy to use the report generator in TestArchitect to show the test results per system requirement: define a report that shows test requirements, and as grouping field specify the “source” field.

Conclusion

This article has outlined that good test design has 3 major components:

  • Creating test cases that directly relate to system requirements
  • Creating test cases that indirectly relate to system requirements
  • Creating test cases that do not directly relate to system requirements

The third item is often overlooked and can be very critical in a successful Test Automation effort.

More Information

  • More information about Action-Based Testing may be found here.
  • More information on TestArchitect may be found here.
Hans Buwalda
Hans Buwalda, CTO of LogiGear, is a pioneer of the Action Based and Soap Opera methodologies of testing and automation, and lead developer of TestArchitect, LogiGear’s keyword-based toolset for software test design, automation and management. He is co-author of Integrated Test Design and Automation, and a frequent speaker at test conferences.

The Related Post

LogiGear Magazine – April 2014 – Test Tool and Automation
Take 5 and test your knowledge on the Software Testing essentials covered in our March 2019 issue. Good luck!
This book isn’t for everyone, but everyone can get some value out of it. What I mean by that rather confusing statement is that folks working in Agile environments will likely want to throw the book across the room while folks in more bureaucratic environments like CMMI or other waterfall environments will likely get a ...
Regardless of your current state of tools, building an effective Continuous Integration suite of significant automated regression tests is the key to moving to a higher level of confidence in today’s development world. In the evolution timeline of software development tools, new tools have recently proliferated. We have all been sold on collaboration, transparency and ...
Has this ever happened to you: You’ve been testing for a while, perhaps building off of a branch, only to find out that, after all of this time, there is something big wrong. It’s a bad build and now you have to go backwards, fix something, and get a new build. Basically, you just wasted ...
This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives. Introduction There are many potential pitfalls to Manual Software Testing, including: Manual Testing is slow and costly. Manual tests do not scale well. Manual Testing is not consistent or repeatable. Lack of training. Testing is difficult ...
As our world continues its digital transformation with excitement in the advancement and convergence of so many technologies- from AI, machine learning, big data and analytics, to device mesh connectivity, nor should we forget VR and AR- 2017 promises to be a year that further transforms the way we work, play and take care of ...
June Issue 2019: Testing the Software Car
Mobile testers need to take a different approach when it comes to Test Automation.
“Testing Applications on the web” – 2nd EditionAuthors: Hung Q. Nguyen, Bob Johnson, Michael HackettPublisher: Wiley; edition (May 16, 2003) This is good book. If you test web apps, you should buy it!, April 20, 2001By Dr. Cem Kaner – Director of Florida Institute of Technology’s Center for Software Testing Education & Research Book Reviews ...
I got some comments on my post “Test Everything all the Time” — most notably people commenting that it’s impossible to test “everything”. I can’t agree more. The intention of the post was to make the point that we need to be able to test “everything we can” all the time. That is, you should ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe