Automation Testing Tools in Full View

In order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.

Taxonomy is a systematic description of tool features and therefore requires proper understanding of those features. Chaotic taxonomy means you do not really grasp the technology, or usage of such tools.

Existing Taxonomies

Software testing lacks standards, and software test automation lacks them almost completely.

  • The section on testing tools in software testing chapter of Wikipedia1 is very confusing–to say the least.
  • ISO/IEC 29119 software testing standard2 is under development and far from complete.
  • Software process standards such as TMMi3 or TPI4 state their tool taxonomy only indirectly–by stating vaguely what types of test tools are required for various maturity levels.
  • Maturity Model for Automated Software Testing (MMAST)5 sounds promising, but is far from satisfactory, and almost totally unknown in software industry.

Last but not least, the most world-known de facto standard is ISTQB: International Software Testing Qualifications Board.6 Its syllabi do offer a relatively comprehensive test tools classification, but it is long from sufficient to be useful in practice.

The categories used there: test management, test execution, debugging and troubleshooting, fault seeding, simulation, static and dynamic analysis, keyword-driven test, performance testing and Web tools7 is more a mishmash of names belonging to different worlds (e.g. is keyword-driven tool something different than test execution tool?) rather than a useful taxonomy.

Tool Taxonomy for Agile Testing

Of course, a good and useful test tool taxonomy is the same for agile and for traditional development methods, but in agile methods you need it more, because product and project requirements change faster.

You cannot know all tools, but if you use a good taxonomy, than you have got a framework that enables you to locate and understand tools fast. Even a tool with a totally confusing name (that means, any tool—for some inexplicable reason both freeware and commercial test tools have names designed to impress and confuse, but never to help and guide the user), you are still able to understand roughly what it is in less than 30 seconds, provided you have the framework.

Using the framework presented below, you are no longer one-tool expert unable to see the context, nor you need be surprised by every new tool you encounter! But of course taxonomy does not replace detailed knowledge of given tools.

Outside the Framework

Some issues that are vital for understanding test tools do not fit nicely into tool taxonomy framework. We describe them shortly in this section.

  • What test activities can be automated? The activities can range from intellectual to creative, and administrative to repeatable—you need to know which are best candidates for automation.
  • Tools for testing products, projects and processes. Even if you seldom talk about project or process testing” (names like “audit” or “status monitoring” are used instead), such activities exist and are very important. In this article, we focus on product testing tools. But where does testing tools belong?
  • Using tools is not yet automation. Tools must be used when human senses are not enough. Our eyes cannot see electrical or radio signals, and our hands cannot produce them. However, if you use a manually operated signal generator and a line listener, it is not yet test automation! This should be kept in mind.
  • General types of test tools. Tools can be classified in a number of ways, but not all of them are useful in practice. However, you must be aware of these less important classifications, too, and not become confused.
  • Dedicated test tools and generic tools used for testing. The border-line is often unclear, but you must not mix up things. For example, a word processor used for writing test case specifications, or a spreadsheet used for managing bug reports are very useful for automating some aspects of testing, but no one would call them “test tools.”
  • Test tools can be software applications (most common usage), mechanical tools, electric or electronic measurement instruments and generators (this includes tools more commonly used for HW testing such as oscilloscopes or logic analyzers), cameras or other tracking instruments (used often for usability testing), chemicals sensors, and many more.
  • Open source, freeware and commercial tools. Licensing schemes for commercial tools can be very different. Tool architecture and usage mode become more varied, too, as more and more cloud (software as a service, SAAS) tools appear.
  • The distinction between emulators, simulators, test environment (even for manual testing), and debuggers, is not always easy to make, nor clear-cut.
  • Tools with different degrees of intrusiveness from coverage measurement tools that instrument (change) the source code to electrical measurement instruments, whose only intrusiveness is probe effect in the form of resistance or inductance
  • The level of tool integration. Increasingly, tools are not used alone but with many other tools as part of an integrated environment for automated testing. This creates many new issues, which are not covered in this article.

It must be kept in mind that real tools often span across many categories of these classifications. Some of the categories can be argued do not belong to testing at all, for example debugging or configuration management.

Action-based Framework

The most obvious and useful classification is dividing tools by what they do. It is comfortable to identify four categories here: test management, test execution, testware preparation and special test tools.
Test Management Tools
Test project management tools

  • Tools for test reporting
  • Change and configuration management tools
  • Incident management tools
  • Tools for build and integration
  • Requirement tracking tools

Test Execution Tools

  • Functionality 1: the ability to start dynamic test (send input data, activate signal, invoke a procedure, press mechanical sensor); tool’s “hands.”
  • Functionality 2: the ability to record actual test result (output data, activated signal, radio signal); tools “eyes.”
  • Functionality 3: the ability to compare actual and expected outcome; tool’s “brains.”

Typical test execution tools have all three functionalities, but there may well exist tools with only two of them or even one. Tools for dynamic analysis do not need any “hands,” and not tools for static testing and static analysis, either.

Testware Preparation Tools

Test data preparation tools: such tools can prepare input data, or pairs input— expected output data for dynamic testing, or can be used for populating databases, and for many other purposes.

  • Test document preparation tools: there exist tools for producing test specifications from requirements or design (model-based testing). Then there is the gray zone of test reporting and incident management tools (that belong as well to the test management tools family), or tools for creating test logs (part of test execution tools).
  • Test script preparation tools:
    • Tools for script programming (with additional option for data-driven testing).
    • Capture tools (typically, as part of capture-replay tools).
    • Creating test scripts from source code (most commonly for unit testing); special cases here are automatic creating of test stubs and test drivers, and creating tests for markup languages (mostly HTML).
    • Keyword-driven script generation: creating scripts from test specifications written in formal language.
  • Special tools: tools for test coverage measurement and for fault injection.

Goal-based Tool Classification

What are the test goals?

Security Testing
To some extent, any test tool can be used for security testing, but there are some specialized tools as well. Some are used for penetration testing–dynamic design and execution of tests addressing some known security problems. But even static code analysis tools often contain a large number of rules for security issues.

Safety Testing
There are few tools used specially for testing system safety. Some of them are hardware tools.

Performance Testing
Performance (load, stress) testing tools are test execution tools with special capabilities for creating big load and measuring performance parameters exactly. They come in many different versions and for various platforms, and their sales make today the biggest part of test tool industry.

Conformance Testing
Various types of dynamic and static test execution tools, that focus on whether system behavior conforms to legal, standard or industry rules.

Tools for Testing Other Quality Attributes
Since at least 50-100 quality attributes can be identified (depending on the classifications), many more tool types can identified. This is not very useful, however.

Platform-based Test Tool Classification

There are thousands of different platforms for software systems, depending on hardware, operating system, API, network type, programming languages and other aspects. Most questions asked about software tools concern this issue, which can hide other, core aspects of test tools. But they are very important in practice. Some sub-classifications here: Web tools (very broad category), language-specific tools, tools for different operating systems (including the growing OS flora for mobile devices—Android, Symbian, iOS…), tools for various HW platforms, tools for testing embedded systems.

Level-based Tool Classification

Bottom-up classification framework: hardware tools, debuggers, tools for static code analysis, unit testing tools and frameworks, tools for system and acceptance testing (including usability).

  1. (all links checked on 12 March 2011)
  4. See Test Process Improvement: A step-by-step guide to structured testing, Tim Koomen and Martin Pol
Bogdan Bereza-Jarocinski
A trainer, quality specialist and owner of VictO (2004), Bereza-Jarocinski co-founded Swedish SAST (1995), Polish SJSI (2003), and Polish SPIN (2006). He is also founder and secretary general of ADPQB (2008). For more information on his academy, visit

The Related Post

When Netflix decided to enter the Android ecosystem, we faced a daunting set of challenges: 1. We wanted to release rapidly (every 6-8 weeks). 2. There were hundreds of Android devices of different shapes, versions, capacities, and specifications which need to playback audio and video. 3. We wanted to keep the team small and happy. ...
With the new year just around the corner, here’s a look at the Test Automation trends that have the potential to dominate. DevOps is being relied upon more than ever. With there being strong Market Drivers for the adoption of DevOps, the need for Test Automation has also never been greater. But what’s next after ...
Identifying which tests to begin with when starting automation is key to driving testing cycle times down and coverage up. So there you are. You’ve done a little research and made the business case to upper management regarding test automation and they bit on the proposal. Surprisingly, they supported you all the way and are extremely ...
Are you frustrated with vendors of test automation tools that do not tell you the whole story about what it takes to automate testing? Are you tired of trying to implement test automation without breaking the bank and without overloading yourself with work? I experienced first-hand why people find test automation difficult, and I developed ...
Based in Alberta, Canada, Jonathan Kohl takes time out of his busy schedule to discuss his views on software testing and automation.
How lagging automotive design principles adversely affect final products. Cars are integrating more and more software with every model year. The ginormous screen introduced by Tesla in their flagship Model S a few years ago was seemingly unrivaled at the time. Nowadays, screens of this size are not only commonplace in vehicles such as the ...
When configured with a Python harness, TestArchitect can be used to automate testing on software for custom hardware Unlike other proprietary and open source tools, that are able to automate only desktop, or mobile, TestArchitect (TA Test) has the ability to test the software that runs on hardware in the following ways: 1. TA can ...
This article was developed from concepts in the book Global Software Test Automation: A Discussion of Software Testing for Executives, by Hung Q. Nguyen, Michael Hacket and Brent K. Whitlock Introduction The top 5 pitfalls encountered by managers employing software Test Automation are: Uncertainty and lack of control Poor scalability and maintainability Low Test Automation ...
One of my current responsibilities is to find ways to automate, as much as practical, the ‘testing’ of the user experience (UX) for complex web-based applications. In my view, full test automation of UX is impractical and probably unwise; however, we can use automation to find potential UX problems, or undesirable effects, even in rich, ...
The huge range of mobile devices used to browse the web now means testing a mobile website before delivery is critical.
Framework: An abstraction in which software providing generic functionality can be selectively changed by additional user written code, thus providing application specific software. A software framework is a universal, reusable software platform used to develop applications, products and solutions. Harness: A collection of software and test data configured to test a program unit by running it under varying conditions and monitoring ...
Mobile testers need to take a different approach when it comes to Test Automation.

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news