March 2007 - The Third Holy Grail of Test Design

By Hans Buwalda, Chief Technology Officer, LogiGear Corporation

Introduction

This is the last in a series of four articles that started with "Key Principles of Test Design". In these four articles I present what I view to be three key principles to make test design successful (the "Holy Grails of Test Design"):

  1. Effective break down of the tests
  2. Right approach per test module
  3. Right level of test specification

If you followed the instructions of the previous articles you should now have a list of well-defined and differentiated "test modules". For each test module you should have "test requirements", and you should know what the test cases are going to be. Now a last crucial step is to write down the test cases as clearly and efficiently as possible.

Writing Test Cases at the Right Level of Abstraction

The challenge at this point, the "Third Holy Grail of Test Design", is to write the test cases at the right level of abstraction:

  • Detailed enough to clearly show the intention and logic of the test case: what is the input, what is verified, etc
  • At the same time hiding as many details as possible that are not relevant for the test

This principle is most clearly visible when you use Action Based TestingT (ABT) or a similar key-word driven approach. In ABT the tests are written as a sequence of actions with arguments. The actions are the basis of the automation. This allows you to "hide" those steps that are not signifi-cant for a test in the implementation of the action.

However, even for manual tests it can make sense to "hide" detailed steps that are not relevant, especially when such details are repeated many times. A common example is logging into the system. Let us say that the manual instruction is:

Enter a user name in the field "User Name", and a password in the field "Pass-word". Then click on the button called "Login".

It is not uncommon to find an instruction like this repeated many times in a set of manual test in-structions. Some disadvantages are:

  • Instructions are repeated over and over again, which can be a lot of work.
  • The test cases are hard to read, because of the needless detail it is difficult to see the forest through the trees.
  • If there are changes in the logon screen of the system under test all the test cases have to be updated (or become outdated).
  • In this example the values that actually are interesting are the user name and password. However, they are not specified, only mentioned implicitly. This means that during test execution the tester has to come up with the values over and over again.

When using actions I would normally write something like this:

user password
logon hans logigear

The values are now explicitly specified, while the actual steps needed to log on are not visible. They are "hidden" in the interpretation of the action "logon". Technically this is a simple step, simi-lar to defining subroutines in a programming language. The important point though is the test de-sign objective of:

  • Showing those details that are relevant for a test, like input values
  • Hiding anything else as much as possible

As another example consider these lines. They click a node in a tree, and check whether an item called "parabola" appears in a list:

window tree tree item path
click tree item main pictures My Projects/Main/Picture 1
window
wait for window main
window list item
check list item exists main picture elements parabola

These fragments are rich in details, explicitly telling us which item in a tree to click, to wait for a window to respond, and to check for an element in a list. Whether or not this is appropriate de-pends on the scope of the test:

  • Was the goal to verify the workings of the "pictures" tree and the "picture elements" list? In that case it is good to show the details of this interaction.
  • If the goal was just to see if "parabola" appears as an element in the "picture 1" picture then the details should be avoided.

In the project where this fragment comes from, the goal was just to verify the contents of "picture 1", and therefore the fragment was too detailed. This was more so because the similar fragments (with other values) appeared in many dozens of places throughout the test set. In such a case it is much better to write something like this:

project picture element
check picture element Main Picture 1 parabola

With this notation the purpose of the check is clearer, the number of lines are reduced and there will be less maintenance when the system under test undergoes changes.

Another category where it is easy to save on details is action arguments. In many cases argu-ments like a "zip code" or "phone number" are not relevant for a test. They are just there to com-plete underlying dialogs. If that is the case, leave the arguments out and make sure the action implementations use suitable default values for them.

Conclusion

Exactly what to show and to hide is not always an easy decision, which is why I have named this the third "holy grail". A crucial note here is that the decision is a test design decision, not an engi-neering one! The purpose of hiding or showing details is not necessarily to make the test as short as possible, like you would when writing a program with subroutines. What you need to do most of all is write clear test cases that:

  • Show explicitly what is relevant for the test allowing the reader to understand the test based solely on the test lines, without having to look into the details of an action
  • Hide those steps and arguments that are not relevant in the scope of the test, to avoid unneeded maintenance, and to make the test easier to read

These types of test design decisions need to come from the testers, not from the automation en-gineers. Automation engineer can, however, play a useful role pointing out to testers what is pos-sible, but ultimately the test design decisions belong to the testers.

Other Recent Articles by This Author