Test Automation Is Not Automatic

Recently while teaching a workshop on Testing Dirty Systems, I uttered this “Randyism” off the top of my head, “Test automation is not automatic.” I realized immediately that I had just concisely stated the problem in making test automation a reality in many organizations.

Most testers know that test automation is not automatic. (Wouldn’t it be great?) However, management many times does not know or accept that reality.

There are some test tools, such as unit test tools, that are practically automatically applied. My remarks in this article are aimed at the capture/playback and scripting tools for test automation.

The issues are that:

  • Not every test can or should be automated.
  • For those tests that can be automated, it takes time and effort to build the automation.
  • For those tests that have been automated, the tests must be maintained.
  • It takes time to lean how to use a tool.
  • It takes effort and planning to implement a test automation framework.

None of these are automatic, even with the best of tools.

Not every test can or should be automated

Think about the things you test that are not very repeatable. Or, they may be prone to constant change. Perhaps the things you test are developed in a technology that has little or no tool support.

Then, there are tests such as user acceptance tests that need people’s evaluation to judge acceptance.

Some tests require creativity and adaptation to perform. You may have to make judgments during the test, which may be too complex to describe in a script. Test automation leverages the mundane testing to give more time and attention to the unique tests.

Your job is to identify the tests that can be automated. (They don’t come labeled!) Then, you must understand the nature of the test, such as the pre-requisites, the steps to perform it, the exceptions, and where to find the expected results. None of this is automatic. It’s all test design and test implementation.

For those tests that can be automated, it takes time and effort to build the automation

It’s one thing to automate a function, but another to design a good test of the function. That’s why capture/playback is so appealing, yet lacking. The issues are: What are you testing in the capture session? Then, how can you extend those tests to add value?

You have to apply approaches like data-driven testing and keyword-driven testing, which take time and effort to understand and implement. These are not “out of the box” deliverables.

For those tests that have been automated, the tests must be maintained

This has been one of the consistent issues in test automation. There are things you can do to ease the maintenance burden, but it doesn’t do away with the issue.

For example, modular and reusable test scripts are very helpful in reducing the number of tests that must be maintained. Still, you must maintain the scripts you have.

This means you must know when the application has changed, what the changes were, and create new tests for those changes. This implies the presence of configuration management and traceability.

It takes time to learn how to use a tool

Of course, the learning curve varies by tool, but the fact remains that a tool with sophisticated features will take some time to learn. The learning curve also varies by person. Some people with deep experience in automation may be able to learn very fast, but when you consider the wider deployment, most people will fall on the lower end of the experience scale.

The time also varies by the approach used to get training. Some people try self-learning which is noble, but typically takes longer than classroom training. Mentoring from an experienced person may be the best approach.

You must assess your organization’s skills and abilities to know if mentoring will help. The best mentor in the world can’t help the person who isn’t ready to learn. Is that another “Randyism?”

It takes effort, money and planning to implement a test automation framework

The framework can take a variety of forms, but the context here is the organizational framework which provides a way to efficiently build and control test automation. Tools are only one-third of the picture. You also need processes with trained and motivated people to make everything work together.

Out of the box, tools are just software. A process framework helps with reuse and the maintenance of test automation.

Frameworks aren’t automatic, either. They must be adapted to fit each situation, therefore they require time, effort and funding to design and implement.

Conclusion

This article is certainly not an in-depth treatment of this topic. Entire books and training courses have been written to address these and other aspects of test automation.

I hope this expands on my simple statement “Test automation is not automatic” to provoke your own thinking on this reality.

For over twenty years now, I have seen a wide variety of test tool companies promote their tools as being effort-free, script-less, or however they choose to position their products. The evidence shows these are often empty claims. Just compare the numbers of people who have been successful in using test automation tools to those who have given up using the tools. It’s about 25% successful to 75% unsuccessful in my research.

I hope this article is an encouragement to those who have tried to implement test automation as well as to those who haven’t. It’s important to go into these projects with your eyes open and expectations at a realistic level. Once you get past the idea that the tools do all the work, you can do the planning and other work needed to increase your chances of success.

Randall Rice
Randy Rice is a thought leading author, speaker and practitioner consultant in the field of software testing and software quality. Rice has worked with organizations worldwide to improve the quality of their information systems and optimize their testing processes.

The Related Post

In order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.
As our world continues its digital transformation with excitement in the advancement and convergence of so many technologies- from AI, machine learning, big data and analytics, to device mesh connectivity, nor should we forget VR and AR- 2017 promises to be a year that further transforms the way we work, play and take care of ...
< Michael Hackett sat down with EA’s Stephen Copp to discuss the world of integrated test platforms.
The 12 Do’s and Don’ts of Test Automation When I started my career as a Software Tester a decade ago, Test Automation was viewed with some skepticism.
Introduction A common issue that I come across in projects is the relationship between test automation and programming. In this article I want to highlight some of the differences that I feel exist between the two.
Source: From I.M.Testy (BJ Rollison’s blog) I just finished reading Implementing Automated Software Testing by E.Dustin, T. Garrett, and B. Gauf and overall this is a good read providing some well thought out arguments for beginning an automation project, and provides strategic perspectives to manage a test automation project. The first chapter made several excellent ...
Test Strategy A test strategy describes how the test effort will reach the quality goals set out by the development team. Sometimes called the test approach, test strategy includes, among other things, the testing objective, methods and techniques of testing and the testing environment.
Even the highest quality organizations have tradeoffs when it comes to their testing coverage. In Japan, Europe, and the United States, automotive manufacturers are aiming to enhance automotive functions by using software; in Japan in particular, Toyota, Nissan, Honda, Mazda, and Subaru are all adding endless amounts of software to their vehicles in the form ...
When Netflix decided to enter the Android ecosystem, we faced a daunting set of challenges: 1. We wanted to release rapidly (every 6-8 weeks). 2. There were hundreds of Android devices of different shapes, versions, capacities, and specifications which need to playback audio and video. 3. We wanted to keep the team small and happy. ...
Identifying which tests to begin with when starting automation is key to driving testing cycle times down and coverage up. So there you are. You’ve done a little research and made the business case to upper management regarding test automation and they bit on the proposal. Surprisingly, they supported you all the way and are extremely ...
Developers of large data-intensive software often notice an interesting — though not surprising — phenomenon: When usage of an application jumps dramatically, components that have operated for months without trouble suddenly develop previously undetected errors. For example, the application may have been installed on a different OS-hardware-DBMS-networking platform, or newly added customers may have account ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe