Avoid Epic Fail. Get Professional Help

Bringing in experts can set you up for automation success.

Test automation isn’t easy when your testing gets beyond a few hundred test cases. Lots of brilliant testers and large organizations have, and continue to struggle with test automation, and not for lack of effort. Everyone understands the value of test automation, but few testing organizations realize its full potential.

A year-long IDT (Innovative Defense Technologies) automated software testing survey showed that automated software testing fails to deliver the expected value for the follow reasons:

  • Lack of time: 37%
  • Lack of expertise: 20%
  • Lack of budget: 17%
  • Tool incompatibility: 11%
  • Other (mix of the above, etc.): 15%

Source: Implementing Automated Software Testing by Elfriede Dustin, Thom Garrett and Bernie Gauf

It’s not surprising that lack of time and lack of expertise are cited as the top reasons test automation fails. Test teams often get a tool thrust on them by non-QA decision makers- from a purchasing group that has no idea about testing, or a CIO who was influenced by a tool salesman, their previous company or, something from a conference. These non-QA people do not understand the coding or scripting skill level needed by the team, or may not realize the tool is harder to use than expected, or worse, the tool does not easily work with the application or system under test. As a result , the automation program languishes and then fails. And it’s the test team that looks bad.

If you’re starting a test automation program for the first time, or are charged with making testing faster and better a lot of the challenges will be the same. Test automation success begins at a high level. This is where the lack of (automation) expertise can cripple a project before it gets started.

Many people have unrealistic expectations about how easy or how quickly a program can ramp up and fail to take into account the impact on test schedules. Commitment to time and training and cooperation with developers must come from the highest levels in the organization. Having an outside expert share experiences and educate managers and directors about the complexities and realities of automation programs can set the right expectations and set you up for success by providing:

  • Strategic consulting— assess your current practices and create the strategy for your automation program.
  • Automation Consulting—develop and prioritize the automation effort and supporting activity specific to your needs.
  • Training— instruction in tools, processes and methods.
  • Coaching—hands-on guidance that helps teams become proficient faster.
  • What follows is where organizations can benefit from bringing in a qualified partner to help with their test automation.
Creating the Automation Strategy

Test automation is software development. It needs to be treated like its own project running parallel to production code development, with its own schedule, resourcing and project management. You need commitment and support from the top of the organization for a significant automation suite build-up.

Bringing in someone with prior success in automation implementation—a consultant—to assess your current practices and develop a custom-tailored plan will avoid making costly mistakes. A well-developed strategic plan will define the testing goal and outline the road map to achieve the goal. It will also establish an understanding of testing/quality assurance across the organization, and prescribe staff development to assure the team will have the necessary skills for test automation.

The strategy will guide other aspects of the automation effort, including tool selection. Typically tools are evaluated in this order: cost, ease of use, and features/functionality. There are low-cost easy to use tools— to a point— but getting the tool to do what is needed may require coding/scripting skills the team may not have. Also support is important. Support runs the gamut, from non-existent with open source tools to an included number of hours with commercial tools. The amount of support required will depend on the skills of the team, and it’s good to have when you run into a wall. Also, over-time, you application is likely to undergo quite a bit of change and having a tool that can be customized may be worth paying extra for. These considerations are generally not obvious to teams with little or no prior automation experience and this is where having a well-defined plan will avoid a costly long-term mistake.

Test Automation

This is the stage where the strategy gets implemented, but before just diving in and starting test automation, there’s a lot of work that needs to be done.

You’ll want to build infrastructure that includes naming conventions, standard or common coding practices, source control, error handling, managing data and reporting.

Teams should anticipate spending time to clean up existing tests before automating them. There are significant differences between tests written for manual execution and tests written for automation. Programmers typically don’t pay attention to “designing for testability”. They’re either unaware of how code changes impact automation or don’t effectively communicate changes. The result is that automation is negatively impacted. Failing to address test design issues will result in a large mass of test code that isn’t effective and requires a lot of time just to maintain. When teams understand good test design for automation it will become apparent there will be tests that should be thrown out and replaced with better written tests.

You will also need to develop and prioritize the automation. Taking 100 manual test cases and trying to create 100 automated tests will always be a bad idea. Not all tests are candidates for automation. If the application under test is not stable, it will impact which and how many tests can be automated. Then too, there will be manual tests that can and should be automated, and a lot that shouldn’t be.

Consultants with experience in successful automation implementations can help with all of these areas and enlisting their help at the beginning will get you started on the right track. Once you’re up and running it’s easy to build on success.

Training and Coaching

Training is the essential first step to gaining test automation expertise. There are a lot of good training resources. They range from free, on-demand recordings to generic-instructor led courses, to customized training. We’re seeing more and more companies opt for customized training as it’s not easy to translate generic concepts to specific applications.

The most important training for any team doing test automation, but especially those in Agile and Continuous Integration (CI) environments and teams doing large, high-volume automation, is learning test design. Up-front test design will determine how efficient tests are to run and how easy they are to maintain. Getting this right is what makes it possible to scale automation.

No matter where you are in your automation process, learning low-maintenance test design will enable you to get the most from automation. Action Based Testing (ABT) is a method of test design that lets you do this. It allows you to create common functions and function libraries that make for efficient test design and simplified maintenance. The limitation to training is that it only provides information. You have to turn the knowledge you receive into application, which can be slow, and you’ll probably make mistakes along the way. Getting hands-on instruction will shorten the learning curve dramatically.

Hands-on coaches are the equivalent of personal trainers for testing. Coaches are able to work 1-on-1, and with a small group on your actual testing projects. They can guide you in the test design and the correct approach specific to the application or system under test, and even sit at a keyboard and walk you through a process. Coaches not only help teams become proficient faster, they are also able to jump in and help resolve complex issues so the project keeps moving as expected.

Outsourcing Sense

Automation will not immediately reduce the test effort. In reality the effort typically increases when automation is undertaken. Building the testing infrastructure takes time and utilizes a lot of resources. Designing and creating new tests will also take time. The additional work load can overextend teams already taxed to keep up with current testing demands. So it’s no surprise that lack of time is the number one reason test automation fails.

Outsourcing some or all of the initial work makes a lot of sense. It will be done by people who have done it before. This saves time and avoids mistakes and gets everything integrated and provides you with properly designed tests from the start. Farming the work out can let you can keep up with existing testing, and then have a smooth transition to the new program. When you factor in everything, most of the time this will provide a better ROI than going the in-house route.

Summary

Test automation isn’t easy, but it can be made a lot easier. Expert consultants can assess the structure, staffing, the corporate quality goals, and provide exactly what you need for an effective test automation program. Going this route is not only smart, it will save time, produce long-term benefits, and most of all avoid epic fail.

Joe Luthy
Joe has over 15 years product marketing experience in technology and Internet services for major companies including AT&T and Verisign.

TestArchitect for Visual Studio is a keyword authoring platform extension designed specifically to enhance coded UI test automation in Visual Studio 2012.

The Related Post

Regardless of your current state of tools, building an effective Continuous Integration suite of significant automated regression tests is the key to moving to a higher level of confidence in today’s development world. In the evolution timeline of software development tools, new tools have recently proliferated. We have all been sold on collaboration, transparency and ...
This book isn’t for everyone, but everyone can get some value out of it. What I mean by that rather confusing statement is that folks working in Agile environments will likely want to throw the book across the room while folks in more bureaucratic environments like CMMI or other waterfall environments will likely get a ...
LogiGear Magazine – October 2010
When it comes to performance testing, be smart about what and how you automate Listen closely to the background hum of any agile shop, and you’ll likely hear this ongoing chant: Automate! Automate! Automate! While automation can be incredibly valuable to the agile process, there are some key things to keep in mind when it ...
As I wrote in various articles, organization is one of the 3 key requisites for successful automated testing, the other two being test design and automation architecture.
I got some comments on my post “Test Everything all the Time” — most notably people commenting that it’s impossible to test “everything”. I can’t agree more. The intention of the post was to make the point that we need to be able to test “everything we can” all the time. That is, you should ...
Introduction In many of the Test Automation projects that we are involved with using our Action-Based Testing methodology, management has expressed a need to relate tests and test results to system requirements. The underlying thought is that automation will create extra possibilities to control the level of compliance to requirements of the system under test. ...
Utility: A program that performs a specific task related to the management of computer functions, resources, or files, as password protection, memory management, virus protection, and file compression. Tool: A program or application that software development teams use to create, debug, maintain, or otherwise support other programs and applications. The term usually refers to programs that can be combined together ...
LogiGear Magazine September Issue 2020: Testing Transformations: Modernizing QA in the SDLC
5 roadblocks in vehicular autonomy that complicate Software Testing Experts in the field have previously referred to air travel as somewhat of a gold standard for autonomous vehicle safety, but after Boeing’s two tragedies, that analogy can no longer be used when talking about self-driving cars. This was after Boeing’s 737 MAX Jets have found ...
From cross-device testing, to regression testing, to load testing, to data-driven testing, check out the types of testing that are suitable for Test Automation. Scene: Interior QA Department. Engineering is preparing for a final product launch with a deadline that is 12 weeks away. In 6 weeks, there will be a 1 week quality gate, ...
“Testing Applications on the web” – 2nd EditionAuthors: Hung Q. Nguyen, Bob Johnson, Michael HackettPublisher: Wiley; edition (May 16, 2003) This is good book. If you test web apps, you should buy it!, April 20, 2001By Dr. Cem Kaner – Director of Florida Institute of Technology’s Center for Software Testing Education & Research Book Reviews ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe