Automation Testing Tools in Full View

In order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.

Taxonomy is a systematic description of tool features and therefore requires proper understanding of those features. Chaotic taxonomy means you do not really grasp the technology, or usage of such tools.

Existing Taxonomies

Software testing lacks standards, and software test automation lacks them almost completely.

  • The section on testing tools in software testing chapter of Wikipedia1 is very confusing–to say the least.
  • ISO/IEC 29119 software testing standard2 is under development and far from complete.
  • Software process standards such as TMMi3 or TPI4 state their tool taxonomy only indirectly–by stating vaguely what types of test tools are required for various maturity levels.
  • Maturity Model for Automated Software Testing (MMAST)5 sounds promising, but is far from satisfactory, and almost totally unknown in software industry.

Last but not least, the most world-known de facto standard is ISTQB: International Software Testing Qualifications Board.6 Its syllabi do offer a relatively comprehensive test tools classification, but it is long from sufficient to be useful in practice.

The categories used there: test management, test execution, debugging and troubleshooting, fault seeding, simulation, static and dynamic analysis, keyword-driven test, performance testing and Web tools7 is more a mishmash of names belonging to different worlds (e.g. is keyword-driven tool something different than test execution tool?) rather than a useful taxonomy.

Tool Taxonomy for Agile Testing

Of course, a good and useful test tool taxonomy is the same for agile and for traditional development methods, but in agile methods you need it more, because product and project requirements change faster.

You cannot know all tools, but if you use a good taxonomy, than you have got a framework that enables you to locate and understand tools fast. Even a tool with a totally confusing name (that means, any tool—for some inexplicable reason both freeware and commercial test tools have names designed to impress and confuse, but never to help and guide the user), you are still able to understand roughly what it is in less than 30 seconds, provided you have the framework.

Using the framework presented below, you are no longer one-tool expert unable to see the context, nor you need be surprised by every new tool you encounter! But of course taxonomy does not replace detailed knowledge of given tools.

Outside the Framework

Some issues that are vital for understanding test tools do not fit nicely into tool taxonomy framework. We describe them shortly in this section.

  • What test activities can be automated? The activities can range from intellectual to creative, and administrative to repeatable—you need to know which are best candidates for automation.
  • Tools for testing products, projects and processes. Even if you seldom talk about project or process testing” (names like “audit” or “status monitoring” are used instead), such activities exist and are very important. In this article, we focus on product testing tools. But where does testing tools belong?
  • Using tools is not yet automation. Tools must be used when human senses are not enough. Our eyes cannot see electrical or radio signals, and our hands cannot produce them. However, if you use a manually operated signal generator and a line listener, it is not yet test automation! This should be kept in mind.
  • General types of test tools. Tools can be classified in a number of ways, but not all of them are useful in practice. However, you must be aware of these less important classifications, too, and not become confused.
  • Dedicated test tools and generic tools used for testing. The border-line is often unclear, but you must not mix up things. For example, a word processor used for writing test case specifications, or a spreadsheet used for managing bug reports are very useful for automating some aspects of testing, but no one would call them “test tools.”
  • Test tools can be software applications (most common usage), mechanical tools, electric or electronic measurement instruments and generators (this includes tools more commonly used for HW testing such as oscilloscopes or logic analyzers), cameras or other tracking instruments (used often for usability testing), chemicals sensors, and many more.
  • Open source, freeware and commercial tools. Licensing schemes for commercial tools can be very different. Tool architecture and usage mode become more varied, too, as more and more cloud (software as a service, SAAS) tools appear.
  • The distinction between emulators, simulators, test environment (even for manual testing), and debuggers, is not always easy to make, nor clear-cut.
  • Tools with different degrees of intrusiveness from coverage measurement tools that instrument (change) the source code to electrical measurement instruments, whose only intrusiveness is probe effect in the form of resistance or inductance
  • The level of tool integration. Increasingly, tools are not used alone but with many other tools as part of an integrated environment for automated testing. This creates many new issues, which are not covered in this article.

It must be kept in mind that real tools often span across many categories of these classifications. Some of the categories can be argued do not belong to testing at all, for example debugging or configuration management.

Action-based Framework

The most obvious and useful classification is dividing tools by what they do. It is comfortable to identify four categories here: test management, test execution, testware preparation and special test tools.
Test Management Tools
Test project management tools

  • Tools for test reporting
  • Change and configuration management tools
  • Incident management tools
  • Tools for build and integration
  • Requirement tracking tools

Test Execution Tools

  • Functionality 1: the ability to start dynamic test (send input data, activate signal, invoke a procedure, press mechanical sensor); tool’s “hands.”
  • Functionality 2: the ability to record actual test result (output data, activated signal, radio signal); tools “eyes.”
  • Functionality 3: the ability to compare actual and expected outcome; tool’s “brains.”

Typical test execution tools have all three functionalities, but there may well exist tools with only two of them or even one. Tools for dynamic analysis do not need any “hands,” and not tools for static testing and static analysis, either.

Testware Preparation Tools

Test data preparation tools: such tools can prepare input data, or pairs input— expected output data for dynamic testing, or can be used for populating databases, and for many other purposes.

  • Test document preparation tools: there exist tools for producing test specifications from requirements or design (model-based testing). Then there is the gray zone of test reporting and incident management tools (that belong as well to the test management tools family), or tools for creating test logs (part of test execution tools).
  • Test script preparation tools:
    • Tools for script programming (with additional option for data-driven testing).
    • Capture tools (typically, as part of capture-replay tools).
    • Creating test scripts from source code (most commonly for unit testing); special cases here are automatic creating of test stubs and test drivers, and creating tests for markup languages (mostly HTML).
    • Keyword-driven script generation: creating scripts from test specifications written in formal language.
  • Special tools: tools for test coverage measurement and for fault injection.

Goal-based Tool Classification

What are the test goals?

Security Testing
To some extent, any test tool can be used for security testing, but there are some specialized tools as well. Some are used for penetration testing–dynamic design and execution of tests addressing some known security problems. But even static code analysis tools often contain a large number of rules for security issues.

Safety Testing
There are few tools used specially for testing system safety. Some of them are hardware tools.

Performance Testing
Performance (load, stress) testing tools are test execution tools with special capabilities for creating big load and measuring performance parameters exactly. They come in many different versions and for various platforms, and their sales make today the biggest part of test tool industry.

Conformance Testing
Various types of dynamic and static test execution tools, that focus on whether system behavior conforms to legal, standard or industry rules.

Tools for Testing Other Quality Attributes
Since at least 50-100 quality attributes can be identified (depending on the classifications), many more tool types can identified. This is not very useful, however.

Platform-based Test Tool Classification

There are thousands of different platforms for software systems, depending on hardware, operating system, API, network type, programming languages and other aspects. Most questions asked about software tools concern this issue, which can hide other, core aspects of test tools. But they are very important in practice. Some sub-classifications here: Web tools (very broad category), language-specific tools, tools for different operating systems (including the growing OS flora for mobile devices—Android, Symbian, iOS…), tools for various HW platforms, tools for testing embedded systems.

Level-based Tool Classification

Bottom-up classification framework: hardware tools, debuggers, tools for static code analysis, unit testing tools and frameworks, tools for system and acceptance testing (including usability).

  1. http://en.wikipedia.org/wiki/Software_testing#Testing_tools (all links checked on 12 March 2011)
  2. http://softwaretestingstandard.org/
  3. https://www.tmmi.org/
  4. See Test Process Improvement: A step-by-step guide to structured testing, Tim Koomen and Martin Pol
  5. http://sqa.fyicenter.com/art/A_Maturity_Model_for_Automated_Software_Testing.html
  6. http://istqb.org/display/ISTQB/Home
Bogdan Bereza-Jarocinski
A trainer, quality specialist and owner of VictO (2004), Bereza-Jarocinski co-founded Swedish SAST (1995), Polish SJSI (2003), and Polish SPIN (2006). He is also founder and secretary general of ADPQB (2008). For more information on his academy, visit www.victo.eu.

The Related Post

Introduction In many of the Test Automation projects that we are involved with using our Action-Based Testing methodology, management has expressed a need to relate tests and test results to system requirements. The underlying thought is that automation will create extra possibilities to control the level of compliance to requirements of the system under test. ...
This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives. Introduction There are many potential pitfalls to Manual Software Testing, including: Manual Testing is slow and costly. Manual tests do not scale well. Manual Testing is not consistent or repeatable. Lack of training. Testing is difficult ...
Bringing in experts can set you up for automation success. Test automation isn’t easy when your testing gets beyond a few hundred test cases. Lots of brilliant testers and large organizations have, and continue to struggle with test automation, and not for lack of effort. Everyone understands the value of test automation, but few testing ...
A short-list of selection criteria and popular automation tools. There are a lot of test automation tools available in the market, from heavy-duty enterprise level tools to quick and dirty playback-and-record tools for browser testing. For anyone just starting their research we’ve put together a short list of requirements and tools to consider.
What is the Automation ROI ticker? The LogiGear Automation Return on Investment (ROI) ticker, the set of colored numbers that you see above the page, shows how much money we presumably save our customers over time by employing test automation as compared to doing those same tests manually, both at the design and execution level.
I feel like I’ve spent most of my career learning how to write good automated tests in an Agile environment. When I downloaded JUnit in the year 2000 it didn’t take long before I was hooked – unit tests for everything in sight. That gratifying green bar is near-instant feedback that everything is going as ...
Introduction Many executives have some very basic questions about Software Testing. These questions address the elements of quality (customer satisfaction) and money (spending the least amount of money to prevent future loss). The basic questions that executive have about Software Testing include: Why care about and spend money on testing? Why should testing be treated ...
When it comes to performance testing, be smart about what and how you automate Listen closely to the background hum of any agile shop, and you’ll likely hear this ongoing chant: Automate! Automate! Automate! While automation can be incredibly valuable to the agile process, there are some key things to keep in mind when it ...
How to do UI test automation with the fewest headaches I’m currently interviewing lots of teams that have implemented acceptance testing for my new book. A majority of those interviewed so far have at some point shot themselves in the foot with UI test automation. After speaking to several people who are about to do ...
We’ve scoured the internet to search for videos that provide a wealth of knowledge about Test Automation. We curated this short-list of videos that cover everything from the basics, to the more advanced, and why Test Automation should be part of part of any software development organization. Automation Testing Tutorial for Beginners This tutorial introduces ...
The huge range of mobile devices used to browse the web now means testing a mobile website before delivery is critical.
The following is a transcript of a May 7, 2008 interview with Hung Q. Nguyen, founder and CEO of LogiGear Corporation and coauthor of the best selling textbook Testing Computer Software. Interviewer: When it comes to software testing, what concerns or issues are you hearing from software developers? Hung Q. Nguyen: The most pressing concern ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe