Automation and Requirements

Introduction

In many of the Test Automation projects that we are involved with using our Action-Based Testing methodology, management has expressed a need to relate tests and test results to system requirements. The underlying thought is that automation will create extra possibilities to control the level of compliance to requirements of the system under test. In this article I will explore some of the aspects and considerations regarding tests and requirements.

System and Test Requirements with Action-Based Testing

Action-Based Testing encourages testers to create “test requirements” as part of the test modules (spreadsheets with the tests to be carried out). The test requirements govern the scope of test cases that are in the test module; in other words, they detail the intention the tester has with the cases described.

For example, a system requirement is typically formulated like “the system should.” A test requirement typically reads as “test if.” As a short hand, we often leave the “test if” out, so a test requirement could read “the system does not accept password shorter than 6 characters,” to be understood as “test if the system does not accept password shorter than 6 characters.”

Now the next question is: how do the test requirements relate to system requirements? In many cases the link is very direct. As illustrated in the example above on minimum password length, the test requirement will relate one-on-one to a system requirement. In other cases a system requirement is a composite statement, which would lead to several test requirements.

For example “the system should enforce that passwords are at least 6 characters in length and contain at least 1 digit,” would lead to (1) “the system does not accept passwords shorter than 6 characters,” and (2) “the system requires a password to have at least 1 digit.”

Good Test Design: Going Beyond System Requirements

Another possibility is that a test requirement specifies that a test be created that does not directly relate to any system requirement. For example, a valid test requirement could state “test with 5 users from different states who are using the same mailbox.” This means the test should create a test case in which the described situation is simulated, but there is very likely no system requirement talking about 5 users.

Test requirements that are unrelated to system requirements illustrate an important consideration regarding system requirements and tests: A test designer should use system requirements as input and make sure all of them are covered in test cases, but a good test designer should go beyond that narrow mission. System requirements mostly focus on single elements of a system under test. A tester should make sure that the elements of the system work well together, and that the system can handle complex situations that the developer might not necessarily have thought about, because they were not specifically addressed in system requirements.

One potential benefit of Automation is to provide information related to system requirements. However, this needs to be carefully organized to be effective. Consider the following picture in which some relationships have been illustrated.

The solid arrows show how system requirements are, directly or indirectly, related to test requirements, which in turn are related to test cases, which in turn are related to test results. A test tool like TestArchitect will give you the possibility to establish relations 1-3 in the above picture as follows:

  1. In the details panel for a test requirement it is possible to register a “source”, this field can contain the identifier of the system requirement to which the test requirement relates.
  2. The same detail panel also shows the test cases in the test modules (after they have been created and the module has been checked in at least once), and checkmarks to indicate which one(s) cover the test requirement.
  3. The relation from the test results to the test case will be shown in the test results. Once the test result is checked in the test case details panel will show the latest result.

To easily obtain the consolidated relation (4) make sure the “source” fields of the test requirements are consistently populated, the test requirements are related to test cases and test run results (of successful runs) are checked in. This will make it easy to use the report generator in TestArchitect to show the test results per system requirement: define a report that shows test requirements, and as grouping field specify the “source” field.

Conclusion

This article has outlined that good test design has 3 major components:

  • Creating test cases that directly relate to system requirements
  • Creating test cases that indirectly relate to system requirements
  • Creating test cases that do not directly relate to system requirements

The third item is often overlooked and can be very critical in a successful Test Automation effort.

More Information

  • More information about Action-Based Testing may be found here.
  • More information on TestArchitect may be found here.
Hans Buwalda
Hans Buwalda, CTO of LogiGear, is a pioneer of the Action Based and Soap Opera methodologies of testing and automation, and lead developer of TestArchitect, LogiGear’s keyword-based toolset for software test design, automation and management. He is co-author of Integrated Test Design and Automation, and a frequent speaker at test conferences.

The Related Post

Automated Testing is a huge part of DevOps, but without human-performed quality assurance testing, you’re increasing the risk of  lower-quality software making it into production.  Automated Testing is an essential DevOps practice to increase organizations’ release cadence and code quality. But there are definitely limits to only using Automated Testing. Without human quality assurance (QA) ...
< Michael Hackett sat down with EA’s Stephen Copp to discuss the world of integrated test platforms.
5 roadblocks in vehicular autonomy that complicate Software Testing Experts in the field have previously referred to air travel as somewhat of a gold standard for autonomous vehicle safety, but after Boeing’s two tragedies, that analogy can no longer be used when talking about self-driving cars. This was after Boeing’s 737 MAX Jets have found ...
An Overview of Four Methods for Systematic Test Design Strategy Many people test, but few people use the well-known black-box and white-box test design techniques. The technique most used, however, seems to be testing randomly chosen valid values, followed by error guessing, exploratory testing and the like. Could it be that the more systematic test ...
Test Strategy A test strategy describes how the test effort will reach the quality goals set out by the development team. Sometimes called the test approach, test strategy includes, among other things, the testing objective, methods and techniques of testing and the testing environment.
LogiGear Magazine – The Big Testing Issue – April 2012
When it comes to performance testing, be smart about what and how you automate Listen closely to the background hum of any agile shop, and you’ll likely hear this ongoing chant: Automate! Automate! Automate! While automation can be incredibly valuable to the agile process, there are some key things to keep in mind when it ...
Learn how to leverage TestArchitect and Selenium for turnkey, Automated Web testing. TestArchitect lets you create, manage, and run web-based automated tests on different types of browsers—using either a WebDriver or non-WebDriver technique. In this article, we will explore employing WebDriver for testing a web-based application with TestArchitect. TestArchitect with WebDriver is a tool for automating ...
How to do UI test automation with the fewest headaches I’m currently interviewing lots of teams that have implemented acceptance testing for my new book. A majority of those interviewed so far have at some point shot themselves in the foot with UI test automation. After speaking to several people who are about to do ...
The Cloud demands that we be as nimble as possible, delivering features and fixes in almost real-time fashion. Both customer and provider rely on software development that can maintain quality while being light on its feet and constantly moving. In addition, Cloud-oriented systems tend to be highly complex and dynamic in structure — more than ...
There is no one recipe to make big testing a big success. It takes planning and careful execution of the various aspects, like test design, infrastructure and organization – a mix that can be different for each situation in which you may find yourself. In writing about big testing, the first question that comes up ...
Identifying which tests to begin with when starting automation is key to driving testing cycle times down and coverage up. So there you are. You’ve done a little research and made the business case to upper management regarding test automation and they bit on the proposal. Surprisingly, they supported you all the way and are extremely ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe