Test Automation: Divide and Conquer

By Hans Buwalda, Chief Technology Officer, LogiGear Corporation

Divide and conquer was a strategy successfully employed by ancient Persian kings against their Greek enemies. It is a strategy that can still be used successfully today. Fundamentally, by dividing something into smaller more manageable pieces (in the case of the ancient Persians, they divided the Greek city states), it becomes much more manageable.

Test automation projects generally have two kinds of problems:

1. Tests lack intelligence. Tests follow functional specifications step by step, and are not really "sharp".
2. Test automation is disappointing. Not many tests get automated, and the tests that are automated are difficult to keep "working".

A major source of the problems is that organizations underestimate the different skill sets involved. To be successful, a team needs many skills, but the two most important are:

1. Designing good tests
2. Automating tests successfully

The root of the problem is that many view the automation of tests as a low tech activity that the testers can take care of on top of their test design efforts. Unfortunately, many test tools on the market encourage this vision by making automation "friendly" with nice looking record and playback features and other support for end users to do their own automation. However, automation is in essence software development - you try to program a computer do something that you do not want to do yourself anymore. As with any software, automated tests tend to be complex and they can break when something unanticipated happens.

To create software, and to make it work, takes specific skills and interests. It takes experience and patience to find the cause of problems. This is even more the case with test automation than it is other software:

• The scale of software is usually pretty large. There are many tests to be done, and if all tests are automated this can amount to a lot of automation.
• Test automation is software that needs to communicate with other software that is not necessarily stable, and can provide often subtle changes with every release. Letting software communicate is among the hardest tasks in IT, as every network engineer can attest.
• Adding to this complexity are the many "moving parts". When something goes wrong, the test can be wrong, the implementation of the test can be wrong, and the system under test might be wrong. There could also be a technology problem with the test tool being unable to correctly address a class in the user interface of the system under test.

These skills and experiences are not part of the profile of most testers. The interest and patience needed to dive into ugly problems in test execution is not common among many testers. The best way for test automation to work effectively is when separate people on a team are tasked with it, and sometimes even specifically hired to do it.

Furthermore, it is a good idea to have cooperation with the development organization. This way the automation engineer can get help and coaching, get information on specific classes, API's and protocols used in the application under test, and give feedback on the testability of it all. For example, automation engineers can negotiate extra hooks for the development team to incorporate into the software to make test automation easier and more stable.

Let us now look at the problem from the reverse angle: automation engineers are usually not well suited for test design. Testing is a profession that is often under-respected. In many cases testers are just regarded as workers who convert system specifications (use cases, requirements, UI mockups, etc) one-to-one into test cases. Good test design is, however, a highly developed art. Testers have to devise situations that make life difficult for the system under test, and express them into test cases that are not too tedious or to hard to automate. Test design also takes its own kind of patience. Testers need to work their way through specifications and find the many things to verify. This is something that automation engineers do not particularly like. Like most programmers, they usually prefer to create a new piece of software than to patiently and methodically test an existing one.

The bottom line is obvious; organize testing activities into at least two separate disciplines: 1) Test design, and 2) Test automation engineering. Some people may be able to work on both tasks, but you should not count on it. You should ask your team members what they like. They will usually diverge into individuals who like testing and individuals who like automation. Or conversely, they will diverge into individuals who hate automation and individuals who hate testing. By doing this you will be able to divide your test automation team into an effective unit with the appropriate people designing tests, and the appropriate people automating the tests.