Test Automation Tool Shopping Guide

A short-list of selection criteria and popular automation tools.

There are a lot of test automation tools available in the market, from heavy-duty enterprise level tools to quick and dirty playback-and-record tools for browser testing. For anyone just starting their research we’ve put together a short list of requirements and tools to consider.

Requirements

Selecting the right tool starts by identifying what you need to achieve. If you only need to automate web browser testing, your requirements will be different than if you’re testing a custom built banking application.

Team Skills

Manual test design and automated test design are vastly different. There is no use investing in a tool the team won’t be able to use effectively. This will also be a factor in determining the level of support required from the tool vendor.

Programming Resources

Some tools require the skills of knowledgeable programmers to implement. Easy to implement tools lessen the need for programming support but may sacrifice features and functionality, so the tradeoff must be weighed carefully.

If the tool supports the code your programmers are using, it’s much easier (and simpler) to get programming support for difficult automation tasks. It also allows for the potential of reusing code from unit tests, which saves time and cost of creating automated tests.

Controls and Objects

Operating systems provide dozens of various controls that can be used in tested applications like: edit boxes, list views, check boxes, menus, buttons and others. The tools you consider should include features that simplify working with controls used in your application.

Operating System Support

The tool should support common operating systems and be easy to update for the latest versions. Commercial tools provide maintenance for this, whereas open-source tools not so much.

Language Support

If you need to test applications against multiple languages, you don’t want to have to create tests in every language. The tools you consider should have multiple language capability.

Customization Support

Specialized and complex applications will likely require customization. If this is the case you will want to focus on tools that can accommodate your needs and even offer support services that can do the customization for you.

Ease of Use

With training and experience most tools are easy to use. Record & playback tools satisfy ease of use, but not a lot else. Instead ease of use being a primary selection criteria, it is better to use it as a comparison between the top choices.

Distributed Testing

If you’re doing large-scale testing you will want to have a tool that allows tests to be executed over multiple machines that can be controlled from a single location.

The Tool Short List

Here’s a list of a subset of test automation tools available. Each tool has pluses and minuses. The best tool for any specific application will need to be determined by conducting a thorough evaluation.

HP® QTP (Quick Test Professional)

HP is the market share leader for software test automation tools. QTP works with most Windows and web-based applications. The latest iteration of the tool is UFT (Unified Functional Testing), which includes QTP for front-end web-based testing, and ST (Service Testing) for back-end Web Services-based testing, as well as mobile testing capability.

Pros

QTP has many included web and windows target automation platforms. There are also numerous integrations to 3rd-party add-ons like Flex, QT, Delphi XE2, Stingray 11.1, Web Dynprop ABAP for NW 7.31, etc., that can be purchased for specific application needs.

Novice and non-technical testers can create basic, low-level automation scripts with the record and playback feature.

QTP is integrated with HP’s Test Management tool, Quality Center (QC), to map QTP-created test cases that can then be executed in QC.

Cons

QTP test cases are heavily reliant on scripting. Test cases created with the record and playback functionality will have to be re-recorded whenever the application under test changes.

QTP only supports VBScript (a readable English-like language).

As test cases scalability grows, test case maintainability becomes a larger part of the test effort.

QTP is a heavy weight on the cost side for both licensing and computing resources. Licensing cost varies depending on the number and type of licenses. Some versions of QTP require extra-cost add ins/plugins to work with DotNet components.

Microsoft® Visual Studio® Coded UI

Visual Studio 2010, included Coded UI, as an integrated approach to get testers and developers on a unified platform. Now on the third rendition, Visual Studio 2013, Coded UI, and Team Foundation Server (TFS) with Microsoft Test Manager (MTM) form the foundation for UI test automation is a software development and application lifecycle management (ALM) environment.

Pros

Microsoft’s Coded UI integrates across Microsoft’s with Team Foundation Server (TFS) and Microsoft Test Manager (MTM).

User actions can be recorded while running a test case using MTM (Microsoft Test Manager). The automated recorded actions can be repeated any number of times after a change is made in a build.

In Visual Studio 2013, Coded UI tests support a variety of technologies, including XAML-based Windows Store apps, IE 11 and SharePoint 2013, in addition to previously supported technologies. It provides logging support for cross-browser tests.

Cons

Test creation is time consuming and Coded UI generates a large amount of C#-only code, even for small UI operations, complicating test maintenance.

Coded UI uses almost all the property-value pairs to identify and locate the UI controls, in reference to the UI MAP, which can affect the performance of the script.

Even if a set of code has been generated for a particular action, Coded UI will regenerate exactly the same code when the test actions are repeated, without leveraging the value of code reusability.

TestArchitectTM for Visual Studio®

This version of TestArchitect is an extension for Visual Studio (2012/2013), providing built-in functionality. The tool uses Action Based Testing (ABT), an advanced keyword method that provides a number of key advantages over Coded UI automation.

Pros

Coding and scripting is replaced with reusable actions for test creation. Built-in, preprogrammed actions simplify test creation.

Tests have high reusability, reducing the amount of time needed to create and execute tests. Test cases can be managed in TFS and executed in MTM.

TestArchitect uses the construct of test modules to organize test cases. This functionality makes it possible to abstract tests to the business-logic level, with objectives or user stories for each test. Tests are aggregated to the same or similar scope, making test flow organization a key focus of test development.

Cons

TestArchitect for Visual Studio only supports platforms that are supported by Coded UI. There is no iOS or Android mobile support.

TestArchitectTM

TestArchitect is the enterprise test automation tool that integrates a test design methodology (ABT), test automation, and test automation life-cycle management (TLM). It addresses the needs of large-scale test automation for complex applications by focusing on test and automation maintainability.

Pros

Actions are used in place of scripts or programming for test creation. A centralized repository allows remote teams to leverage reusable test assets.

Allows distributed test execution on multiple physical or virtual machines. Includes a variations feature that allows a testing against multiple languages with a single test.

A programmable harness is provided to support customization for proprietary or third-party platforms and controls.

Cons

TestArchitect is not ideal for small-scale test creation. The systematic approach to test design and test flow is best suited for high-volume testing.

A steeper initial learning curve is required to take advantage of the ABT test methodology, than recording and scripting.

SmartBear TestComplete®

TestComplete is targeted at small to midsize test teams. It includes keyword testing to make the tool usable for a wider audience of testers. Smartbear also offers a SaaS version of the tool.

Pros

TestComplete’s record-replay feature makes it simple to record and automate basic tests. The tool works well for small to mid-size teams looking to automate a portion of their software testing.

The tool has an attractive price and there are a number of add-ons that can be purchased to extend basic capability.

Cons

Record and play-back tests require a significant amount of maintenance when the application under test is changing rapidly.

Hand-scripting test cases are an alternative to record and playback, but hand-scripting can be difficult or for novice or non-technical testers.

Micro Focus SilkTest®

SilkTest is used for automating functional tests for enterprise-level software applications. SilkTest’s automation capabilities make it a consideration for regression, cross-platform and localization testing across a broad set of application technologies.

Pros

SilkTest supports a range of application platforms, including AJAX/Flex/Web 2.0, Java, .NET,client/server system. It also provides cross-browser support, without test script modification.

The embedded VB .Net scripting language enables test customization and automation through a visual-based or editor-based editing environment.

Cons

SilkTest is limited to supporting only Windows applications and only through Internet Explorer and Mozilla browsers on Windows.

SilkTest use 4Test scripting language, an object oriented language similar to C++, which requires a programming-centric learning curve for novice and non-technical testers.

Selenium

The Selenium Suite is a popular open source tool exclusively for testing web-based applications through browsers. The Selenium suite offers the choice of the Selenium IDE or Selenium RC (Resource Control) (previous version) or Selenium WebDriver (newer version), depending on the level of testing needed.

Pros

The open-source software has a zero up-front cost, and focuses on easy to create tests for web browser testing.

It has a record and playback feature for creating tests without scripting in a test domain-specific language (Selenese)

User actions can be recorded in popular languages like Java, C#, Perl and Ruby.

Cons

Selenium is for automating browser testing only. Tests are challenging to scale and maintain for large-scale projects.

Selenium tests need to start the browser and stop the browser between each test. Tests can sometimes break due to long page load times.

Support and functionality updates are only available through the open source community. Updates and maintenance require strong programming skills.

Summary

There are a lot of choices when it comes to test automation. Every tool has its plusses and minuses. Which one, or which ones, is the right choice for your organization will have many dependencies. Starting by thoroughly evaluating your needs and the team (and their skills) that will be working with and managing the tool will help narrow down your choices. Getting it right will provide a significant boost to your testing effort.

LogiGear Corporation
LogiGear Corporation provides global solutions for software testing, and offers public and corporate software testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast, cost-effective results. Since 1994, LogiGear has worked with Fortune 500 companies to early-stage start-ups in, creating unique solutions to meet their clients’ needs. With facilities in the US and Viet Nam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

The Related Post

For this interview, we talked to Greg Wester, Senior Member Technical Staff, Craig Jennings, Senior Director, Quality Engineering and Ritu Ganguly, QE Director at Salesforce. Salesforce.com is a cloud-based enterprise software company specializing in software as a service (SaaS). Best known for its Customer Relationship Management (CRM) product, it was ranked number 27 in Fortune’s 100 ...
This book isn’t for everyone, but everyone can get some value out of it. What I mean by that rather confusing statement is that folks working in Agile environments will likely want to throw the book across the room while folks in more bureaucratic environments like CMMI or other waterfall environments will likely get a ...
Utility: A program that performs a specific task related to the management of computer functions, resources, or files, as password protection, memory management, virus protection, and file compression. Tool: A program or application that software development teams use to create, debug, maintain, or otherwise support other programs and applications. The term usually refers to programs that can be combined together ...
There are few topics in quality assurance testing that cause as much confusion as smoke testing versus sanity testing. The two names would seem to describe very different practices— and they do! But people still get them confused, since the distinction is somewhat subtle.
Even the highest quality organizations have tradeoffs when it comes to their testing coverage. In Japan, Europe, and the United States, automotive manufacturers are aiming to enhance automotive functions by using software; in Japan in particular, Toyota, Nissan, Honda, Mazda, and Subaru are all adding endless amounts of software to their vehicles in the form ...
When Netflix decided to enter the Android ecosystem, we faced a daunting set of challenges: 1. We wanted to release rapidly (every 6-8 weeks). 2. There were hundreds of Android devices of different shapes, versions, capacities, and specifications which need to playback audio and video. 3. We wanted to keep the team small and happy. ...
An automation framework is a way to organize your code in meaningful manner so that any person who is working with you can understand what each file contains. Automation frameworks differ based on how you organize your code – it can be organized based on your data, so that any person who wants to use ...
From automotive Software Testing standards, testing techniques, and process, this article is an in-depth guide for those looking to transfer their existing skills to this exciting industry. For the Software Car, autonomous driving gets most of the hype, but most overlook the fact that there is so much more to Software Testing for the automotive ...
Many organizations rely on HP Quality Center to design test plans and track test results. TestArchitect’s Quality Center integration makes working with QC as easy as pie. TestArchitect (TA) is a three-in-one tool for Test Management, Test Development, and Test Automation. Users can create and manage test assets, execute tests, track and analyze test results, ...
Recently while teaching a workshop on Testing Dirty Systems, I uttered this “Randyism” off the top of my head, “Test automation is not automatic.” I realized immediately that I had just concisely stated the problem in making test automation a reality in many organizations. Most testers know that test automation is not automatic. (Wouldn’t it be great?) However, ...
The growing complexity of the Human-Machine Interface (HMI) in cars offers traditional testers an opportunity to capitalize on their strengths. The human-machine interface (HMI) is nothing new. Any user interface including a graphical user interface (GUI) falls under the category of human-machine interface. HMI is more commonly being used to mean a view into the ...
Having the right Test Automation plan helps bridge gaps and fragmentations in the complex mobile environment. Figuring out the best Test Automation plan is one of the biggest frustrations for today’s digital teams. Organizations struggle to develop cross-platform Test Automation that can fit with their Continuous Integration cadence, their regression cycles and other elements of ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe