May 2007 - Automation Selection Criteria - Picking the "Right" Candidates (Part 2 of 3)

By Bob Galen, Founder RGalen Consulting Group, LLC

Introduction

Taking a key from the anti-patterns mentioned in the first article in this series, there are some general selection patterns that should normally be used to govern thinking around selecting good test cases. In this article, the second in this series, we will examine these practices with the view towards establishing them within your own automation efforts.

Selection Criteria - Patterns

Low Hanging Fruit

As was alluded to in the anti-patterns part of this article series, this pattern is focused on generating momentum and not in picking the best, most impact producing automation targets. It usually happens in the beginning of an automation effort or when you have received a major new version of your automation tools. In any event, low hanging fruit test cases are just that - they are usually small, localized to well understood and stable functional components and generally quick to implement.

It is usually best to focus new automation developers on these cases for learning curve and to measure their capabilities before assigning them more challenging work. A final important part of the pattern is making successes visible - both internally within the testing team and across the organization. So showing early success off a bit is an important part of the pattern.

Fleshing Out the Infrastructure

All good automation efforts establish an infrastructural layer that wraps their tools in order to support efficient automation development. Usually this is focused towards implementing automation development templates, standards, naming conventions and other guidelines that enforce consistent practices and procedures for development. It often also involves writing some wrapper code that programmatically reinforces the consistency and serves to reduce startup time and learning curve.

These practices need to be verified and debugged, so selecting good candidates to flesh out and verify the infrastructure becomes important. Usually these candidates are distributed along the architectural boundaries of the Application Under Test (AUT) - so that the infrastructure is verified to support all aspects of the application environment.

Minimizing Rework

There are clearly two factors that can drastically affect your efforts and influence rework - requirement stability and application stability. While you can not always control either, you can choose when to automate and thus mitigate impacts from the two. Experience shows the easier to control is requirement stability. In this case, you simply want to wait until there is sign-off or agreement on requirements. Another indicator here is that the development team is beginning construction. Normally this is a good indication that if the requirements are stable enough for development they might be stable enough for automaton.

Application stability is a tougher factor to control. Here it is usually best to wait to trigger automation development until after the team has at least had a chance to exercise application features manually. If manual execution is impossible, for example with an API, than any demonstrated testing and positive results will serve as a trigger for automation development.

Of course a good strategy in both of these cases is to defer your automation construction until after the specific application release is deployed. In this case, you take both aspects virtually out of play and drastically reduce rework risk.

Driving with Value

At a fundamental level, automation value is driven by the coverage it provides and time it saves contrasted against the time it takes to automate the test. The potential lifetime of the automation then is a strong factor in determining its value. Many teams write automation for very transient application features. They will run it only a few times and then the application morphs in another direction.

They are stuck either taking the maintenance hit to change the automation or retiring it. In both cases, the value of that particular piece of automation was impacted. Value is also determined by the customer value associated with the specific feature. How central is it to their business needs and frequency of use?

You should consider the return potential for every automation decision. More times than not, you are looking to automation capabilities that are stable - so that the automation has a high lifetime before maintenance is required. These features also need to be of high customer value - leading towards heavy interest in repeated testing cycles.

Planning - Consider Complexity & Packaging Tempo

One of the more important factors to consider is simply how hard things are to implement. More product development teams should consider this when developing their products. If you have 100 test cases that you want to automate and they are all exceedingly complex, then you are somewhat stacking the deck against yourself for delivery.

What is preferable to do is create a more normally distributed set of test cases, say 25% very complex, 50% moderately complex, and 25% relatively straightforward, when developing an automation release package. Create a goal for the package that represents one of the overall patterns in this section. For example, after a first automation Low Hanging Fruit package development it may make sense to go for a cycle time or speed focused goal of Project Impact - Consider Speed release. This way you immediately start leveraging automation impact on the project. Regardless of the approach, you should develop a packaging strategy that uses goals and short iterations to create a tempo for your automation release.

You also never want to underestimate hand-off activities nor level of effort.

Project Impact - Consider Speed

Testers always want to maximize the impact they have on their projects, which is usually focused on risk mitigation. Automation also has the capacity to drastically impact testing speed - so examining how quickly a package of automation can be pulled together for product execution becomes a very compelling view. In this case, identifying test cases that take the longest time to execute manually and automating them can have a drastic effect on testing cycle time.

Of course you need to consider all aspects of time when picking these candidates, including test setup time, data preparation time, execution time, and results analysis time. Clearly you do not want to automate the smallest or fastest tests first if you want to have a high impact on cycle times within your project.

Project Impact - Consider Risk

One of the most powerful applications of automation is as a risk mitigation factor within your projects. From that perspective, you may want to choose candidates that align with some of the more challenging areas of the application. Areas where you know the development team will struggle and where you can make a difference. This may mean that you take less on - in order to have a larger impact on the products' maturity and stabilization.

One key practice to consider here is the Pareto Principle or 80:20 rule. Basically it states that 80% of the bugs (or risk) will be driven by 20% of the overall application. If you can identify and isolate these 20% areas and apply your automation efforts towards them, you will be well on your way towards effectively mitigating project risk, and your developers and project managers will love you for it!

Project Impact - Consider Maintenance Effort

One of the truly hidden properties behind developing test automation is that it is indeed a software project. Given that, it also succumbs to maintenance issues both in the short term, as you try and automate a volatile application, and long term as the application evolves and matures. Nearly 20-30% of automation development time is spent in handling traditional maintenance level activities, even when using well architected automation solutions. It can potentially get much worse than this.

One way to handle maintenance work is to steal a technique from the development lifecycle and have periodic maintenance releases of automation. Accrue maintenance work for these sorts of releases and time them to coincide with possible slack time within projects. This also allows you to better understand, isolate, measure and communicate the costs as well.

Conclusion

Clearly you do not need to consider all of these patterns in every selection decision, and they all have different weights depending upon your specific context. The primary goal is to create a breadth to your analysis and consideration that truly targets test case selection towards making a visible and fundamental impact for each of your projects. It should clearly be seen as a differentiator when your broader product team discusses project success factors.

The next article in this series will deal with prioritization and criteria adjustment factors to allow you to be nimble over time.

About the Author

Bob Galen is the author of Software Endgames (Dorset House, 2005). He is also the founder of The RGalen Consulting Group, a technical consulting company that focuses on agility and pragmatism. Bob has over 25 years of experience as a software developer, tester, project manager and leader.

Other Articles by this Author