March 2007 - Developing a Business Case for Test Automation (Part 1 of 3)
By Bob Galen, Founder RGalen Consulting Group, LLCIntroduction Most presentations relating to establishing an automation business case center on hard return on investment (ROI) factors such as cost saving and cost cutting as the primary value proposition. While these are certainly important factors, a business case built on these factors alone is far too narrow. There are other factors that broadly define a compelling test automation business case. Simply put - it isn't all about the dollars and cents. In fact, the real differentiating factors are actually never monetary in nature. A focus solely on hard ROI factors can lead stakeholders to consider trimming and compressing the test team - something that is rarely a good idea. The purpose of this series of articles is to share the breadth of factors that make for a compelling business case for test automation. Such a business case will not only drive organizational support, but it will create some excitement and expectations within an organization's test team to do a great job in automating testing. There are three articles that will present the breadth of factors that make up a good business case for test automation. These articles will discuss:
These articles will lead an organization toward constructing a broad and effective Business Case for their automation efforts, one that will not only compel stakeholders, but equally compel the product development teams. Assessing the Automation Climate An often forgotten step in business case development is actually figuring out the compelling drivers for it within an organization. Quite often teams dive into developing test automation with a specific view towards time and cost savings. While time and cost savings can surely be a result, over zealously looking to cut costs, even though it seems to be the right response, should be cautioned against. Rather, an organization should take a broader investigative approach to sorting out their core motivations - one that assesses their unique context or climate and then forms a set of motivating factors that are targeted towards solving their specific challenges. Following are the most common assessment areas to examine when reviewing the automation business climate: Application Characteristics Quite often targeted application(s) for automation determine the level and capacity of automation efforts. For example, GUI or web based applications are often easy targets for test automation, so the relevance and opportunity for these applications might be very clear and broad. However, what if GUI applications have high data quantity dependencies in their backend databases? Data that takes literally weeks to populate from a variety of sources and is renewed on a monthly basis would clearly impact automation efforts. Other sorts of applications, for example embedded medical systems, system software, or SOA applications will also affect automation viability, breadth, and strategies. Another factor that clouds application support are the ability of various tools to support the languages, libraries, tools, and third party components used to build an organization's applications. Many tools have difficulty interacting with certain controls and other components. While this sounds only inconvenient, it can actually inhibit testing of specific features that leverage these components. In fact, it is not uncommon for component or control incompatibility to virtually halt large areas of automation efforts. The key here is to thoroughly evaluate the organization's tool sets against their applications. Not simply performing an evaluation, but planning for a longer pilot to truly ascertain the actual support level and to identify problem areas. Any real limitations or suspected risks that will impact test automation development should be surfaced and factored into the business case. Team Capabilities An often underestimated assessment factor is the depth of experience and capabilities of an organization's team. There are several factors that come into play. First is the familiarity the team has with basic software and automation development skills. Has the team built tools and software before? Have they built test automation? Do they understand the difference between writing automated test cases and developing an automation architecture and framework that supports efficient automation development and execution? Usually an organization will also need some sort of architectural capabilities - an individual or individuals with deep technical skills in the product technologies and with the automation tools. The organization will rely heavily on this person or people to set the stage for their automation efforts. Often teams do not have the requisite depth in development experience to start an automation program on their own, and a few courses in tools and techniques will not be enough to get the job done. In these cases, the organization might want to identify a consulting partner who can supplement their team's experience gaps and to truly jump-start their efforts. An organization should not underestimate the length of time that such help will be needed, as these are longer-term core competencies that are needed for ongoing success. Budget Constraints Succinctly put - today's production automation tools are not inexpensive. In fact, there are many layers to the cost model beyond simply purchasing the tools. There will be licensing costs associated with the tool and simple start-up training. Often, the vendor supplied training will need to be supplanted with additional courses or consultant mentors to truly bring the organization's team up to speed. The organization should factor in reoccurring maintenance costs at least for the first year, as well as execution costs? Often the tools used for development will need to be replicated in order to physically run the automation. This may require virtually two sets of licenses, to support development as well as run-time requirements. This also leads into additional costs for infrastructure (physical space, power, networking support, and basic software assets) and hardware to support the automation environment. It is not atypical for every "seat" of automation to drive 2-5 systems that are supporting automation development and execution. The organization should perform an exhaustive analysis of tool needs for the first 18 months of life including training, startup licenses for development and execution, first year maintenance costs, and infrastructure and hardware costs. These should be included in the budget setup for the automation business case. If the automation effort will scale over this time, these increased needs will need to be included as well. The key point here is to shake out the clear and hidden costs to ensure that stakeholders fully understand and buy into realistic and complete upfront automation development costs. Leadership Understanding The final and most important point is to assess the understanding of the organization's leadership of automation and gain an understanding of their overall expectations. For understanding, the organization should assess whether the leadership team understands that developing test automation is equivalent to developing software. That it requires the same development processes. That it will require ongoing maintenance and support of the infrastructure and toolset. With regard to expectations, often the leadership team does not have a clear understanding of the time needed to implement automation as well as the amount of time that will elapse before it begins to have an impact on the organization's projects and starts to generate an ROI. Does management understand that a baseline set of tools and infrastructure must be established before the organization can begin automating test cases? Do they understand that this can often take quite a bit of time and effort - perhaps even months to establish. Do they also understand that automation follows a prescribed Software Development Lifecycle - so that its introduction needs to be properly planned for execution? Finally, do they understand that it needs to integrate with the development project stream and that often creates priority conflicts and skew in the automations' ability to impact existing product line development? If there are any gaps in these areas, they should be mapped as risks that need to be tackled along the way in the business case. It is very common for stakeholders to initially have unreasonable expectations for automation development that need to be re-mapped back to reality. In many cases, the business case needs to be adjusted to compensate for this. Wrapping up the Assessment After the data gathering phase, some time should be taken to write up the findings. For the assessment, the focus is less about capturing all of the data, and more about finding the key forces or goals that will be driving automation efforts. It is important to identify forces that will block automation success so that they may be dealt with either within the business case or by taking actions across the organization. Force Field Analysis can be an effective tool to use as a way to wrap the assessment and also prepare the business case. The forces for become the most compelling outcomes from executing test automation, while the forces against are mostly mapped back to your assessment results. Conclusion Once the automation climate is effectively assessed, the organization will need to move on to determine what the "hard" and "soft" ROI goals are for the automation effort. Both of these will be covered in more detail in the following two articles in this series. About the Author Bob Galen is the author of Software Endgames (Dorset House, 2005). He is also the founder of The RGalen Consulting Group, a technical consulting company that focuses on agility and pragmatism. Bob has over 25 years of experience as a software developer, tester, project manager and leader. |
