“Software is complex but I’m tired of finding bug after bug that a 5th grader would have turned in. Virtually every technical product these days includes a lot of software. It’s rare that an engineer can write nearly perfect code. Methodical and thorough testing of software is the key to quality products that do what the user expects. Read this book to learn what you need to do!”

Global Software Test Automation is the first book to offer software testing strategies and tactics for executives. Written by executives and endorsed by executives, it is also the first to offer a practical business case for effective test automation, as part of the innovative new approach to software testing: Global Test Automation — a proven solution, backed by case studies, that leverages both test automation and offshoring to meet your organization’s quality goals.

The following is a review from Scott Barber, Chief Technologist at PerfTestPlus.

“Happy About Global Software Test Automation: A Discussion of Software Testing for Executives is an absolute must read for any executive in a company that develops, customizes or implements software. For years, software testing has been notoriously under valued and misunderstood by corporate executives. While leading software testers have been trying to get their message to executives from the bottom up, they have been largely unsuccessful. This book has the potential to change that.

With this book, all it takes is one business trip and you’ll be able to engage in risk and ROI based planning to minimize many of the challenges and expenses your company faces related to software through the efficient and effective application and management of software testing.” ■

Chapter 6: Strategies and Tactics for Global Test Automation

In this chapter, you will learn the following:

• The benefits of Global Test Automation

• The seven-step process of developing a Global Test Automation strategy and roadmap

Introduction

In the previous chapters, we have discussed software testing and a number of pitfalls associated with software testing. In particular, we have discussed manual software testing, test automation, and outsourcing/offshoring of software testing. We have also presented a number of suggestions to improve the results in each of these areas, responding to the pitfalls you may experience. In this chapter, we present a comprehensive methodology to address the pitfalls and create a successful test effort.
This methodology entails an array of powerful strategies and tactics for Global Test Automation that creates successful outcomes by intelligently combining manual software testing, test automation, and outsourcing/offshoring of software testing.

What is Global Test Automation (GTA)?

We can all agree that software testing is necessary. We need to test software to be sure that it performs the functions it is designed to perform, under the conditions in which it will be deployed, and in a responsive and user-satisfying manner. We also know that manual software testing, software test automation, and outsourcing/offshoring all inter-relate yet have distinct characteristics with unique issues that need to be addressed. By understanding their pitfalls and suggestions for improvement in these
areas, you will gain a fuller understanding of how Global Test Automation can create a holistic solution for your organization’s testing needs.

Software testing takes time and costs money. As an executive, you want to have a strategy that will provide the needed results while saving both time and money. The 2 by 2 chart in Figure 8 shows strategies for saving time and saving money. But how can you save both time and money? That is where the Global Test Automation strategy comes in. It saves time by speeding up the test process, saves money, and provides the needed results.

An Exercise for the Reader

The first step in establishing a test strategy and methodology is to assess where your organization is currently in its test strategy. To help you internalize the material in this chapter and apply it to your organization, we have provided this exercise for you to begin to evaluate your organization’s current test strategy. Please consider the following questions and answer them for yourself in regards to your organization.

1. How much, in terms of percentage to revenue and/or development dollars respectively, do you budget for software testing?

2. What is your percentage of automated tests versus manual tests?

3. What are the three things that you want to change in your testing strategies to optimize the quality of your released product?

4. What are the three things that you want to change in your testing strategies to optimize the ROI on your test spending?

An Illustration of the Issues

After working on this exercise, you see how important visibility is in making management decisions regarding testing. Visibility gives you the power to make the right choices for the strategic direction of your company. You need visibility into the test process to set the best strategic directions for testing, as well. The right quantitative measurements, test metrics, can give you that visibility. Automation alone won’t
necessarily provide you with that visibility, but it can help. Automation isn’t a silver bullet, but it’s a part of the solution.

Global Test Automation is an integration of the latest test automation methodologies and technologies with global resource strategies to fully capitalize on the speed and cost advantages of best practices in automation and global sourcing. That is a mouthful, so let us break it down into the critical aspects and discuss each one independently.

Global Test Automation is the integrated solution for:

• Software test automation

• Outsource/offshore software testing

• Global team management

The main problems with manual testing are that it is too slow, too expensive, and does not scale. Software test automation can address these issues, if strategically and skillfully applied. However, so long as applications are meant for human end users, test automation will never entirely replace the need for human testers. No matter how sophisticated test automation tools become, they will never be as good as human testers at finding bugs in an application. Human testers will instantly notice subtle bugs that are almost never detected by test automation, particularly usability bugs. Automated test tools cannot “follow their instincts” to uncover bugs using exploratory and ad-hoc testing techniques. By freeing manual testers from having to execute repetitive, mundane tests, properly deployed test automation enables them to focus on using their creativity, knowledge, and instincts to discover more important bugs.

Strategy Formulation

The steps in creating an effective test automation strategy are to assess your testing capability, define a good methodology, select the proper tools to implement this methodology, and put people in place with the proper skills and training to successfully implement the defined test methodology using these tools. Common problems in test automation include its potentially high cost and inability to obtain the desired ROI due to a lack of high productivity and anticipated savings. Scalability, reusability, visibility,
and maintainability can be problematic.

The Global Test Automation strategy addresses these issues in the four phases of test automation: deployment, production, execution, and maintenance. By providing visibility, the GTA strategy utilizing the Action-Based Testing (ABT) methodology greatly improves manageability, and consequently improves the test coverage and test quality. It also addresses scalability and reusability. These four benefits of GTA
(scalability, reusability, visibility, and maintainability) combine to effect high productivity (see Figure 7 in Chapter 4).

The main problems with outsourcing and offshoring software testing include communications problems due to cultural issues and time zone differences and incorrect skill sets. The GTA strategy provides a structured approach that addresses these problems, including a combination of clear, repeatable and manageable processes, appropriate training, powerful tools, and effective management procedures.

The strategy of Global Test Automation is central to its success. The strategy provides a bridge between the problems of outdated manual testing, attempts to address the speed problems with test automation, and attempts to address the cost problems with outsourcing and offshoring of software testing, with the desired end result being an integrated Global Test Automation strategy that achieves both time and cost savings with the desired testing benefits. Global Test Automation makes use of a combination of powerful test automation technology for distributed teams for speed, world-wide resources for cost control, and best practices in management of software testing.

There are seven steps to establishing a successful Global Test Automation strategy in your organization. The steps are identified below:

1. Assess your testing needs.

2. Align your test process.

3. Leverage automation.

4. Minimize costs and risks of global resources.

5. Select the right tools.

6. Secure/develop competency.

7. Measure, set goals, and optimize.
We will describe each of these steps in the following sections.

We call the strategy development methodology for Global Test Automation “SP3™”, which is named after the first initials of each of the critical elements in the strategy development process. Figure 10 graphically illustrates this concept:

A strategy to integrate people, practice, and process for success—the graphic describes that test strategy consists of inter-relationships between people, process, and practice. Process incorporates the lifecycle of testing. People incorporates the combination of skill sets, communication, and morale. Practice involves methodologies and tools.

To obtain a free PDF copy of the book, please email logigearmagazine@logigear.com

Do testers have to write code?

For years, whenever someone asked me if I thought testers had to know how to write code, I’ve responded: “Of course not.”

The way I see it, test automation is inherently a programming activity. Anyone tasked with automating tests should know how to program.

But not all testers are doing test automation.

Testers who specialize in exploratory testing bring a different and extremely valuable set of skills to the party. Good testers have critical thinking, analytical, and investigative skills. They understand risk and have a deep understanding where bugs tend to hide. They have excellent communication skills. Most good testers have some measure of technical skill such as system administration, databases, networks, etc. that lends itself to gray box testing. But some of the very best testers I’ve worked with could not have coded their way out of a For Loop.

So unless they’re automating tests, I don’t think that testers should be required to have programming skills.

Increasingly I’ve been hearing that Agile teams expect all the testers to know how to write code. That made me curious. Has the job market really shifted so much for testers with the rise of Agile? Do testers really have to know how to code in order to get ahead?

My assistant Melinda and I set out to find the answer to those questions. Because we are committed to releasing only accurate data, we ended up doing this study three times. The first time we did it, I lost confidence in how we were counting job ads, so we threw the data out entirely. The second time we did it, I published some early results showing that more than 75% of the ads requested programming skills. But then we found problems with our data, so I didn’t publish the rest of the results and we started over. Third time’s a charm, right?

So here, finally, are the results of our third attempt at quantifying the demand for programming skills in testers. This time I have confidence in our data.

We surveyed 187 job ads seeking Software Testers or QA from across the 29 states in the US posted between August 25 and October 16, 2010.

The vast majority of our data came from Craigslist (102 job ads) and LinkedIn (69 job ads); the rest came from a small handful of miscellaneous sites.

The jobs represent positions open at 166 distinct, identifiable companies. The greatest number of positions posted by any single company was two.

Although we tried to avoid a geographic bias, there is a bias in our data toward the West Coast. (We ended up with 84 job listings in California alone.) This might reflect where the jobs are, or it could be because we did this research in California so it affected our search results. I’m not sure.

In order to make sure that our data reflected real jobs with real employers we screened out any jobs advertised by agencies. That might bias our sample toward companies that care enough to source their own candidates, but it prevents our data from being polluted by duplicate listings and fake job ads used to garner a pool of candidates.

Based on our sample, here’s what we found: out of the 187 jobs we sampled, 112 jobs indicate that programming of some kind is required; an additional 39 jobs indicate that programming is a nice skill to have. That’s just over 80% of test jobs requesting programming skill.

Just in case that sample was skewed by including test automation jobs, I removed the 23 jobs with titles like “Test Automation Engineer” or “Developer in Test.” Of the remaining 164 jobs, 93 required programming and 37 said it’s a nice to have. That’s still 79% of QA/Test jobs requesting programming.

It’s important to understand how we counted the job ads.

We counted any job ad as requiring programming skills if the ad required experience or knowledge of a specific programming language or stated that the job duties required using a programming language. Similarly, we counted a job ad as requesting programming skills if it indicated that knowledge of a specific language was a nice to have.

The job ads mentioned all sorts of things that different people might, or might not, count as a programming language. For our purposes, we counted SQL and shell/batch scripting as programming languages. A tiny number of job ads (6) indicated that they required programming without listing a specific language by listing broad experience requirements like “Application development in multiple coding languages.” Those counted too.

The bottom line is that our numbers indicate approximately 80% of the job ads you’d find if searching for jobs in Software QA or Test are asking for programming skills.

Regardless of my personal beliefs, that data suggests that anyone who is serious about a career in testing would do well to pick up at least one programming language.

So which programming languages should you pick up? Here were the top 10 mentioned programming languages (including both required and nice-to-haves):

• SQL or relational database skills (84)
• Java, including J2EE and EJBs (52)
• Perl (44)
• Python (39)
• C/C++ (30)
• Shell Scripting (27) note: an additional 4 mentioned batch files.
• JavaScript (24)
• C# (23)
• .NET including VB.NET and ASP.NET but not C# (19)
• Ruby (9)

This data makes it pretty clear to me that at a minimum, professional testers need to know SQL. I will admit that I was a little sad to see that only 9 of the job ads mentioned Ruby. Oh well.

In addition, there were three categories of technical skills that aren’t really programming languages but that came up so often that they’re worth calling out:

• 31 ads mentioned XML
• 28 ads mentioned general Web Development skills including HTTP/HTTPS, HTML, CSS, and XPATH
• 17 ads mentioned Web Services or referenced SOAP and XSL/XSLT

We considered test automation technologies separately from programming languages. Out of our sample, 27 job ads said that they require knowledge of test automation tools and an additional 50 ads said that test automation tool knowledge is a nice to have. (As a side note, I find it fascinating that 80% of the ads requested programming skills, but only about half that number mentioned test automation. I’m not sure if there’s anything significant there, but I find it fascinating nonetheless.)

The top test automation technologies were:

• Selenium, including SeleniumRC (31)
• QTP (19)
• XUnit frameworks such as JUnit, NUnit, TestNG, etc. (14)
• LoadRunner (11)
• JMeter (7)
• Winrunner (7)
• SilkTest (6)
• SilkPerformer (4)
• Visual Studio/TFS (4)
• Watir or Watin (4)
• Eggplant (2)
• Fitnesse (2)

Two things stood out to me about that tools list.

First, the number one requested tool is open source. Overall, of the number of test automation tool mentions, more than half are for free or open source tools. I’ve been saying for a while that the commercial test automation tool vendors ought to be nervous. I believe that this data backs me up. The revolution I predicted in 2006 is well under way and Selenium has emerged a winner.

Second, I was surprised at the number of ads mentioning WinRunner: it’s an end-of-lifed product.

My personal opinion (not supported by research) is that this is probably because companies that had made a heavy investment in WinRunner just were not in a position to tear out all their automated tests simply because HP/Mercury decided not to support their tool of choice. Editorializing for a moment: I think that shows yet another problem with closed source commercial products. Selenium can’t ever be end-of-lifed: as long as there is a single user out there, that user will have access to the source and be able to make whatever changes they need.

But I digress.

As long as we were looking at job ads, Melinda and I decided to look into the pay rates that these jobs offered. Only 15 of the ads mentioned pay, and the pay levels were all over the map.

Four of the jobs had pay ranges in the $10-$14/hr range. All four of those positions were part time or temporary contracts. None of the ads required any particular technical skills. They’re entry-level button-pushing positions.

The remaining 11 positions ranged from $40K/year at the low end to $130K/year at the high end. There just are not enough data points to draw any real conclusions related to salary other than what you might expect: jobs in major technology centers (e.g. Massachusetts and California) tend to pay more. If you want more information about salaries and positions, I highly recommend spelunking through the salary data available from the Bureau of Labor Statistics.

And finally I was wondering how many of the positions referred to Agile. The answer was 55 of the job ads.

Even more interesting, of those 55 ads, 49 requested programming skills. So while 80% of all ads requested programming skills, almost 90% of the ads that explicitly referenced Agile did. I don’t think there’s enough data available to draw any firm conclusions about whether the rise of Agile means that more and more testers are expected to know how to write code. But I certainly think it’s interesting.

So, that concludes our fun little romp through 187 job listings. I realize that you might have more questions than I can answer. If you want to analyze the data for yourself, you can find the raw data here.■

Elisabeth Hendrickson founded her company as Quality Tree Consulting in 1997 to provide training and consulting in software quality and testing. She incorporated the company as Quality Tree Software, Inc. in 1998.  In 2003, Elisabeth became involved with the Agile community. In 2005 she became a Certified Scrum Master and in 2006 she joined the board of directors for the Agile Alliance.  You can follow her blog at http://testobsessed.com/ or visit her company site Quality Tree Consulting at  http://www.qualitytree.com/. To read the original post, visit http://testobsessed.com/?s=testers+have+to+write+code%3F

A trainer, quality specialist and owner of VictO (2004), Bereza-Jarocinski co-founded Swedish SAST (1995), Polish SJSI (2003), and Polish SPIN (2006). He is also founder and secretary general of ADPQB (2008). For more information on his academy, visit www.victo.eu.

In  order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.

Taxonomy is a systematic description of tool features and therefore requires proper understanding of those features. Chaotic taxonomy means you do not really grasp the technology, or usage of such tools.

Existing Taxonomies

Software testing lacks standards, and software test automation lacks them almost completely.

• The section on testing tools in software testing chapter of Wikipedia¹ is very confusing–to say the least.
• ISO/IEC 29119 software testing standard² is under development and far from complete.
• Software process standards such as TMMi³ or TPI⁴ state their tool taxonomy only indirectly–by stating vaguely what types of test tools are required for various maturity levels.
• Maturity Model for Automated Software Testing (MMAST)⁵ sounds promising, but is far from satisfactory, and almost totally unknown in software industry.

Last but not least, the most world-known de facto standard is ISTQB: International Software Testing Qualifications Board.⁶ Its syllabi do offer a relatively comprehensive test tools classification, but it is long from sufficient to be useful in practice.

The categories used there: test management, test execution, debugging and troubleshooting, fault seeding, simulation, static and dynamic analysis, keyword-driven test, performance testing and Web tools⁷ is more a mishmash of names belonging to different worlds (e.g. is keyword-driven tool something different than test execution tool?) rather than a useful taxonomy.

Tool Taxonomy for Agile Testing

Of course, a good and useful test tool taxonomy is the same for agile and for traditional development methods, but in agile methods you need it more, because product and project requirements change faster.

You cannot know all tools, but if you use a good taxonomy, than you have got a framework that enables you to locate and understand tools fast. Even a tool with a totally confusing name (that means, any tool—for some inexplicable reason both freeware and commercial test tools have names designed to impress and confuse, but never to help and guide the user), you are still able to understand roughly what it is in less than 30 seconds, provided you have the framework.

Using the framework presented below, you are no longer one-tool expert unable to see the context, nor you need be surprised by every new tool you encounter! But of course taxonomy does not replace detailed knowledge of given tools.

Outside  the  Framework

Some issues that are vital for understanding test tools do not fit nicely into tool taxonomy framework. We describe them shortly in this section.

• What test activities can be automated? The activities can range from intellectual to creative, and administrative to repeatable—you need to know which are best candidates for automation.
• Tools for testing products, projects and processes. Even if you seldom talk about project or process testing” (names like “audit” or “status monitoring” are used instead), such activities exist and are very important. In this article, we focus on product testing tools. But where does testing tools belong?
• Using tools is not yet automation. Tools must be used when human senses are not enough. Our eyes cannot see electrical or radio signals, and our hands cannot produce them. However, if you use a manually operated signal generator and a line listener, it is not yet test automation! This should be kept in mind.
• General types of test tools. Tools can be classified in a number of ways, but not all of them are useful in practice. However, you must be aware of these less important classifications, too, and not become confused.
• Dedicated test tools and generic tools used for testing. The border-line is often unclear, but you must not mix up things. For example, a word processor used for writing test case specifications, or a spreadsheet  used for managing bug reports are very useful for automating some aspects of testing, but no one would call them “test tools.”
• Test tools can be software applications (most common usage), mechanical tools, electric or electronic measurement instruments and generators (this includes tools more commonly used for HW testing such as oscilloscopes or logic analyzers), cameras or other tracking instruments (used often for usability testing), chemicals sensors, and many more.
• Open source, freeware and commercial tools. Licensing schemes for commercial tools can be very different. Tool architecture and usage mode become more varied, too, as more and more cloud (software as a service, SAAS) tools appear.
• The distinction between emulators, simulators, test environment (even for manual testing), and debuggers, is not always easy to make, nor clear-cut.
• Tools with different degrees of intrusiveness from coverage measurement tools that instrument (change) the source code to electrical measurement instruments, whose only intrusiveness is probe effect in the form of resistance or inductance
• The level of tool integration. Increasingly, tools are not used alone but with many other tools as part of an integrated environment for automated testing. This creates many new issues, which are not covered in this article.

It must be kept in mind that real tools often span across many categories of these classifications. Some of the categories can be argued do not belong to testing at all, for example debugging or configuration management.

Action-based  Framework

The most obvious and useful classification is dividing tools by what they do. It is comfortable to identify four categories here: test management, test execution, testware preparation and special test tools.
Test Management Tools
Test project management tools

• Tools for test reporting
• Change and configuration management  tools
• Incident management tools
• Tools for build and integration
• Requirement tracking tools

Test Execution Tools

• Functionality 1: the ability to start dynamic test (send input data, activate signal, invoke a procedure, press mechanical sensor); tool’s “hands.”
• Functionality 2: the ability to record actual test result (output data, activated signal, radio signal); tools “eyes.”
• Functionality 3: the ability to compare actual and expected outcome; tool’s “brains.”

Typical test execution tools have all three functionalities, but there may well exist tools with only two of them or even one. Tools for dynamic analysis do not need any “hands,” and not tools for static testing and static analysis, either.

Testware Preparation Tools

Test data preparation tools: such tools can prepare input data, or pairs input— expected output data for dynamic testing, or can be used for populating databases, and for many other purposes.

• Test document preparation tools: there exist tools for producing test specifications from requirements or design (model-based testing). Then there is the gray zone of test reporting and incident management tools (that belong as well to the test management tools family), or tools for creating test logs (part of test execution tools).
• Test script preparation tools:
  • Tools for script programming (with additional option for data-driven testing).
  • Capture tools (typically, as part of capture-replay tools).
  • Creating test scripts from source code (most commonly for unit testing); special cases here are automatic creating of test stubs and test drivers, and creating tests for markup languages (mostly HTML).
  • Keyword-driven script generation: creating scripts from test specifications written in formal language.
• Special tools: tools for test coverage measurement and for fault injection.

Goal-based Tool Classification

What are the test goals?

Security Testing
To some extent, any test tool can be used for security testing, but there are some specialized tools as well. Some are used for penetration testing–dynamic design and execution of tests addressing some known security problems. But even static code analysis tools often contain a large number of rules for security issues.

Safety Testing
There are few tools used specially for testing system safety. Some of them are hardware tools.

Performance Testing
Performance (load, stress) testing tools are test execution tools with special capabilities for creating big load and measuring performance parameters exactly. They come in many different versions and for various platforms, and their sales make today the biggest part of test tool industry.

Conformance Testing
Various types of dynamic and static test execution tools, that focus on whether system behavior conforms to legal, standard or industry rules.

Tools for Testing Other Quality Attributes
Since at least 50-100 quality attributes can be identified (depending on the classifications), many more tool types can identified. This is not very useful, however.

Platform-based Test Tool Classification

There are thousands of different platforms for software systems, depending on hardware, operating system, API, network type, programming languages and other aspects. Most questions asked about software tools concern this issue, which can hide other, core aspects of test tools. But they are very important in practice. Some sub-classifications here: Web tools (very broad category), language-specific tools, tools for different operating systems (including the growing OS flora for mobile devices—Android, Symbian, iOS…), tools for various HW platforms, tools for testing embedded systems.

Level-based Tool Classification

Bottom-up classification framework: hardware tools, debuggers, tools for static code analysis, unit testing tools and frameworks, tools for system and acceptance testing (including usability). ■

1. http://en.wikipedia.org/wiki/Software_testing#Testing_tools (all links checked on 12 March 2011)
2. http://softwaretestingstandard.org/
3. http://www.tmmifoundation.org/html/tmmiorg.html
4. See Test Process Improvement: A step-by-step guide to structured testing, Tim Koomen and Martin Pol
5. http://sqa.fyicenter.com/art/A_Maturity_Model_for_Automated_Software_Testing.html
6. http://istqb.org/display/ISTQB/Home

Cloud computing, when it is done well, provides a reliable and single point of access for users. Consistent, positive user experience sells the service, and rigorous testing assures a quality experience. In order to produce reliable, effective results for users of many walks of life, exacting software testing standards must be met.

In a series of articles, LogiGear is identifying four fundamental requirements by which software testing in the Cloud can uphold these standards:

Four Fundamental Requirements for Successful Testing in the Cloud:

1. Constantly Test Everything – the Right Way
2. Know What’s Where – and Prove It
3. Define Your Paradigm
4. Don’t Underestimate the Importance of Positive User Experience

In this issue’s article, we address:

Requirement 03: Define your Paradigm

Today’s companies increasingly find that they are in an ever more competitive market, especially in the drive to implement more robust, capable and pioneering Cloud-based products and services. Product delivery times are decreasing, customers demand higher and higher levels of product quality, and failure to deliver within the customer’s expectations can be swiftly punished with whole scale product abandonment and erection of barriers to market reentry.

Companies who are leading creative efforts to address these volatile areas recognize that quality must be uncompromised, and indeed surpassed in every way.

Adopting a testing paradigm that is designed specifically for the requirements of Cloud-computing is a fundamental requirement for the new standards of quality being set by customer-driven demand.

1. Evaluate Your Current Testing Paradigm

Each quality or development team has its own frame of reference by which it identifies the need for testing overall, and the specific kinds of testing required. This frame of reference, or paradigm, often reflects the higher organizational approach to quality assurance and the role that testing is perceived to occupy in customer satisfaction and loyalty.

Cloud computing introduces a need for a new paradigm – one that incorporates the 24/7 testing-on-the-fly required for successful Software as a Service implementation and other Cloud-based products.

Some organizations may find their current approach to testing adapts relatively well for the “test everything all the time” model. However, most will find it necessary to reassess not just their approach toward testing, but the fundamental principles that underlie that approach – ones they’ve inherited from traditional testing models.

In the world of software testing, all the players used to start in pretty much the same place: manual testing. As the industry matured, more elegant testing solutions – record and playback, automated or scripted testing, and action-based testing – emerged over time. Each new approach to testing successively increased our testing efficiencies, reporting efficiencies and ability to handle larger and larger test loads, lending leverage to the leading few. Conducting a Current-State Assessment can give you the necessary insight into what works and doesn’t work in your current testing paradigm when it comes to testing in the Cloud.

Conducting a Current-State Assessment

Most companies have tried a variety of testing approaches, some manual, some scripted, and maybe even some attempts at automation. It is important for a firm to conduct an honest self-assessment about their current testing footprint, capabilities, limitations, and opportunities for improvement.

Many times firm have a belief about their current testing approach and are surprised to find that those beliefs, albeit hopeful, are not fully rooted in practice. The following questions will help thoroughly evaluate your testing paradigm.

• Testing methodology – Is it largely manual or some combination of methods? Are those methods consistent?
• Testing environment – Where are the applications located? Where is the data located? What are the network and performance constructs?
• People resources and skill sets – What is the competency makeup of your current test team? What are the skill sets required for moving forward?
• Time-to-deliver constraints – What are the barriers to testing efficiency? What are the efforts that take more time than others and why?
• Reporting and progress visibility – Do you have consistent visibility into your testing status? Are you surprised when testing efforts are late? Are statuses verbal, or based on actual metrics?

2. Define Paradigm Requirements that Meet Standards for Cloud-based Technologies

Achieving your organization’s testing goals, in whole or in part, requires planning and mapping out your “testing in the Cloud” strategy and its requirements.

Clarifying Cloud Architecture

An initial understanding of the different types of clouds and what their high-level architectures offer will inform the kind of testing paradigm you will need to establish.

Sam Johnston’s integrated picture depicts the three cloud types:

• Remote, Public or External Cloud – Public clouds are sometimes referred to as “Regular” cloud computing. Completely separate from a user’s desktop or the corporate network that they belong to, public clouds offer a pay-per-use service model because the user is leveraging outside compute resources for the particular service they are seeking.This approach offers economies of scale, but their shared infrastructure model can raise concerns about configuration restrictions, adequate security, and service-levels (available uptime). These concerns might make you think twice about subjecting sensitive data that is subject to compliance or safe harbor regulations.Because “public” clouds are typically made available via the public internet they may be free or inexpensive to use. A well known example of a public cloud is Amazon EC2 which is available for use by the general public.
• Internal or Private Cloud – Private cloud computing extends the same infrastructure concepts firms already have in their data centers. The motivation for private clouds appears to be to resolve security and availability concerns inherent in the public cloud paradigm.As such, private clouds seemingly are not burdened by network bandwidth, availability issues or potential security risks that may be associated with public clouds. However, this thinking belies the very intent of cloud computing which is predicated on hardware-software extensibility, dramatic reduction in infrastructure costs, and an elimination of the management concerns governing private networks.
• Mixed or Hybrid Cloud – Many of the leading engineering thinkers in the industry suggest that the most workable cloud computing approach is the “hybrid” approach. The hybrid solution combines the best of the Public and Private Cloud paradigms.Considering that some applications may be too complex or too sensitive to risk operating from a public cloud it makes sense for a firm to protect those application and data assets within the construct of a private cloud where they have total control.Less sensitive applications and data can be migrated to a public cloud freeing compute resources that can be repurposed for the complex applications that need to stay home.The hybrid approach does sound like the best of both worlds. It makes sense from a technology and economic standpoint. It allows for control, flexibility and growth. The trick to managing hybrid clouds comes when you consider spikes in demand.When demand spikes pummel the performance of your applications located within the private cloud -and you need additional computing power (such as is experienced by web-based news media when critical events occur) -you will need to develop a management policy that can be responsible for when to reach out to the public cloud for those additional resources.

Defining the Future-State

Envisioning the results of your testing transformation requires solid understanding of your organization’s business goals and objectives, the Cloud computing paradigms that may help your testing effort contribute to those goals, and development of a sound plan to move in the new direction.

When documenting your planned future state, address each of these categories:

• Architectural – Consider the different Cloud paradigms as they pertain to your business model, goals and objectives, and application and data sensitivity.
• Organization – Ensure organizational review and consensus of new Cloud testing direction as it pertains to business goals and objectives and priorities.
• Financial – Define the benefits, know where the real costs lie, and define the budget.
• Implementation – Adopt an incremental improvement approach, and choose the correct tools and partners.
• Monitor and measure – Develop a consistent set of metrics for measuring and monitoring your new foray into testing in the Cloud.

Other organizational goals and objectives that are important to include:

• Provide sufficient support for distributed test teams
• Increase your return-on-testing-investment
• Significantly decrease time-to-market
• Optimize the reusability of tests and test automation
• Improve test output and coverage
• Enhance the motivation of your testing staff
• Increase managerial control over quality and testing
• Have better visibility into quality and testing effectiveness

While setting your testing goals, consider that in the LogiGear white paper “The 5% Solution,” Hans Buwalda, LogiGear’s CTO, posits that “with good test automation you can have more tests and execute them any time you need to, thus significantly improving the development cycles of your project.”

With the repeatable tests under automation, he says, testers are free to work on more sophisticated testing as well as testing focused on new features and functions increasing the overall quality of the delivered result.

Buwalda establishes the 5% Goals:

• No more than 5% of structured test cases should be tested manually
• No more than 5% of testing efforts should be spent in the automation process

Conversely, the five percent goals suggest that 95% of your tests should be automated and 95% of your testing efforts should be spent on emerging functionality and higher complexity testing. Other organizational goals fall into some pretty predictable categories.

3. Apply Proven Methods and Technology

Implementing the changes necessary to adopt a new testing paradigm will bear out how well you’ve defined your requirements. Therefore, selecting well-matched testing tools and specific architectural approaches can have a dramatic impact on the results of your paradigm shift.

Architectural Approaches

If you are going to perform testing in the cloud it is important to understand the different architectural approaches that are available and the unique advantages that each approach offers. First, let’s start out with a quick list of the six basic cloud architectures or relationships.

Cloud Accessible Application

This category of cloud computing technology focuses on applications that are entirely externally hosted at a cloud services application vendor. No installation of software is required on the user desktop. The application functionality is utilized through a web browser and the data is stored on the application host’s servers. Salesforce.com is a good example of this type of application service.

Cloud-based Client Services

Think of a device that is completely useless without a connection to internet services (like the iPhone) and you will have an understanding of what cloud-based client services are all about. Netbooks are another emerging computing approach predicated on utilizing more cloud-based applications and services using a smaller, less powerful laptop computer.

Scalable Infrastructure and Computational Arrays

Imagine a sudden spike in demand for products you want to sell. Good news, right? But let’s image that your computer room server just can’t handle the additional load. You lose money with every customer who abandons their purchase. With the right cloud infrastructure provider relationship those worries might be a thing of the past. Vendors are increasing building expanding infrastructure architectural offers to support this very problem. Good examples to keep in mind are Amazon and Sun Microsystems.

Application Development and Deployment Platforms

Picture the scenario that you have developed a ‘killer’ application that you know everyone is going to want but you can’t deal with the cost and complexity of buying and managing the server and network equipment to manage it. Web hosting services are a good example of this approach allowing individuals and organizations to provide their own web site accessible via the web. Still other platforms allow for remote applications development and deployment.

Utility Services

MapQuest is popular example of a cloud-based service utility. It is completely inaccessible and unusable if you are not connected to the web. Other examples include, PayPal and Yahoo search.

Data Storage, Duplication and Backup

There are a number of variations of data services available via the cloud. Simple data storage allows for a pay-as-you-increase sort of service allowing ever increasing disk utilization as needed. Other services offer data duplication and mirroring, and data backup.

Conclusion

In this paper, we’ve presented the need to “define your paradigm” as a necessary prerequisite for pursuing a strategy for “testing-in-the-cloud”. We began with the common-sense directive of evaluating your current testing approach and setup. We then reviewed the different cloud types, their focus, and their typical use. Additionally, we offered the business objectives most firms would consider as foundation for making a move to a Cloud-based testing paradigm. Lastly, we presented the six common cloud computing paradigms, the industry standards that have emerged to support each, and introduced examples of services in each category.

Your testing-in-the-Cloud strategy should consider all of these approaches to determine what will work best for your organization. Your success is will be determined by a disciplined approach and thorough implementation. Happy testing!

Test organizations continue to undergo rapid transformation as demands grow for testing efficiencies. Functional test automation is often seen as a way to increase the overall efficiency of functional and system tests. How can a test organization stage itself for functional test automation before an investment in test automation has even been made? Further, how can you continue to harvest the returns from your test design paradigm once the test automation investment has been made? In this article we will discuss the factors in selecting a test design paradigm that expedites functional test automation. We will recommend a test design paradigm and illustrate how this could be applied to both commercial and open-source automation solutions. Finally, we will discuss how to leverage the appropriate test design paradigm once automation has been implemented in both an agile (adaptive) and waterfall (predictive) system development lifecycle (SDLC).

Test design – selection criteria

The test design selection criteria should be grounded in the fundamental goals of any functional automation initiative. Let us assume the selected test automaton tool shall enable end-users to author, maintain and execute automated test cases in a web-enabled, shareable environment. Furthermore, the test automation tool shall support test case design, automation and execution “best practices” as defined by the test organization. To harvest the maximum return from both test design and test automation the test design paradigm must support:

• Manual test case design, execution and reporting
• Automated test case design, execution and reporting
• Data-driven manual and automated test cases
• Reuse of test case “steps” or “components”
• Efficient maintenance of manual and automated test cases

Test design – recommended paradigm

One paradigm that has been gaining momentum under several guises in the last few years is keyword-based test design. I have stated in previous articles that “The keyword concept is founded on the premise that the discrete functional business events that make up any application can be described using a short text description (keyword) and associated parameter value pairs (arguments). By designing keywords to describe discrete functional business events the testers begin to build up a common library of keywords that can be used to create keyword test cases. This is really a process of creating a language (keywords) to describe a sequence of events within the application (test case).”

The Keyword concept is not a silver bullet but it does present a design medium that leads to both effective test case design and ease of automation. Keywords present the opportunity to design test cases in a fashion that supports our previous test design selection criteria. It does not guarantee that these test cases will be effective but it certainly presents the greatest opportunity for success. Leveraging a test design paradigm that is modular and reusable paves the road for long term automation – not only that, it moves most of the maintenance to a higher level of abstraction: the keyword. The keyword name should be a shorthand description of what actions the keyword performs. The keyword name should begin with the action being performed followed by the functional entity followed by descriptive text (if required). Here are several common examples:

• Logon User – Logon User
• Enter Customer Name – Enter Customer Name
• Enter Customer Address – Enter Customer Address
• Validate Customer Name – Validate Customer Name
• Select Customer Record – Select Customer Record

Test design – keyword application

Keyword test case design begins as an itemized list of the test cases to be constructed–usually as a set of named test cases. The internal structure of each test case is then constructed using existing (or new) keywords. Once the design is complete, the appropriate test data (input and results) can be added. Testing the keyword test case design involves executing the test case against the application or applications being tested.
At first glance this does not appear to be any different than any other method for test case design but there are significant differences between keyword test case design and any freehand / textual approach to test case design. Keyword test case designs are:

• Consistent – the same keyword is used to describe the business event every time
• Data Driven – the keyword contains the data required to perform the test step
• Self Documenting – the keyword description contains the designers’ intent
• Maintainable – with consistency comes maintainability
• Automation — supports automation with little or no design transformation (rewrite)

Test design – adaption based on development/testing paradigm

There are two primary development and testing approaches being used by development organizations today: adaptive (agile) and predictive (waterfall/cascade). Both approaches certainly have their proponents–though the increasingly adaptive (agile) system development lifecycles are gaining precedence. The question becomes how does this affect the test design paradigm? The answer appears to be that it really does not affect the test design paradigm but it does affect the timing.

Predictive (waterfall/cascade) development lifecycles can be supported by a straight-forward design, build, execute and maintain test design paradigm that may later support automation. Eventually, one would expect the predictive testing team to design, build, execute, maintain and automate their test case inventory. This could be accomplished using both Tier 1 commercial automation tools and open source automation tools. As long as the automation tools support modular based design (functions) and data driven testing (test data sources) keyword-based automation can be supported–the most significant difference being the time and effort required to implement the testing framework. Adaptive (agile) development lifecycles come in several flavors–some support immediate keyword-based functional test design and automation while others do not. Agile test driven development (TDD) using FitNesse™, a testing framework which requires instrumentation by and collaboration with the development team, certainly supports keyword-based test case design and automation. Other agile paradigms only support instrumentation at the unit test level or not at all; i.e. a separate keyword-based test case design and automation toolset must be used. The challenge for non-TDD agile becomes designing, building, executing and maintaining functional tests within the context of a two to four week sprint. The solution is a combination of technique and timing. For the immediate changes in the current sprint consider using exploratory testers and an itemized list of test cases with little (if any) content–basically a high-level check list. Once the software for a sprint has migrated to and existed in production for at least one sprint, a traditional set of regression test cases can be constructed using keywords. This separates the challenge into sprint-related testing and regression testing.

Test design is the single biggest contributor to success in software testing. Not only can good test design result in good coverage, it is also a major contributor to efficiency. The principle of test design should be “lean and mean.” The tests should be of a manageable size and at the same time complete and aggressive enough to find bugs before a system or system update is released.

Test design is also a major factor for success in test automation. This is not that intuitive. Like many others, I initially also thought that successful automation is an issue of good programming or even “buying the right tool.” Finding that test design is the main driving force for automation success is something that I had to learn over the years-often the hard way.

What I have found is that there are three main goals that need to be achieved in test design. I like to characterize them as the “Three Holy Grails of Test Design” – a metaphor based on the stories of King Arthur and the Round Table as the three goals as the three goals are difficult to reach mimicking thestruggle King Arthur’s knights experienced in search of the Holy Grail. This article will introduce the three “grails” to look for in test design. In subsequent articles of this series I go into more detail about each of the goals.

The terminology in this article and the three subsequent articles are based on Action Based Testing (ABT), LogiGear’s method for testing and test automation. You can read more about the ABT methodology on the LogiGear web site. In ABT test cases are organized into spreadsheets which are called “test modules.” Within the test modules the tests are described as a sequences of “test lines,” each starting in the A column with an “action,” while the other columns contain arguments. The automation in ABT does not focus on automating test cases, but on automating individual actions, which can be re-used as often as necessary.

The Three Goals for Test Design

The three most important goals for test design are:

1. Effective breakdown of the tests

The first step is to breakdown the tests into manageable pieces, which in ABT we call “test modules.” At this point in the process we are not yet describing test cases; we simply place test cases its its appropriate “chapters.” A break down is good if each of the resulting test modules has a clearly defined and well-focused scope which is differentiated from the other modules. The scope of a test module subsequently determines what its test cases should look like.

2. Right approach per test module

Once the break down is done each individual test module becomes a mini-project. Based on the scope of a test module we need to determine what approach to take to develop the test module. By approach I mean the choice of testing techniques used to build the test cases (like boundary analysis, decision tables, etc.), and who should get involved to create and/or assess the tests. For example, a test module aimed at testing the premium calculation of insurance policies might need the involvement of an actuarial department.

3. Right level of test specification

This third goal is deciding where you can win or lose most of the maintainability of automated tests. When creating a test case try to specify only the high-level details that are relevant for the test. For example, from the end-user perspective “login” or “change customer phone number” is one action; it is not necessary to specify any low-level details such as clicks and inputs. These low-level details should be “hidden” at this time in separate, reusable automation functions common to all tests. This makes a test more concise and readable, but most of all it helps maintain the test since low-level details left out will not have to be changed one-by-one in every single test if the underlying system undergoes changes. The low-level details can then be re-specified (or have their automation revised) only once and reused many times in all tests.

In ABT this third principle is visible in the level of the actions to be used in a test module. For example, in an insurance company database, we would write tests using only “high-level” actions like “create policy” and”check premium,” while in a test of a dialog you could use a “low level” action like “click” to see if you can click the OK button.

Conclusion

Regardless of the method you choose, simply spending some time thinking about good test design before writing the first test case will have a very high payback down the line, both in the quality and the efficiency of the tests.

Combinatorial testing can help detect problems like this early in the testing life cycle. The key insight underlying this method is that not every parameter contributes to every failure and most failures are triggered by a single parameter value or interactions between a relatively small number of parameters. To detect interaction failures, software developers often use “pairwise testing”, in which all possible pairs of parameter values are covered by at least one test. Its effectiveness is based on the observation that software failures often involve interactions between parameters. For example, a router may be observed to fail only for a particular protocol when packet volume exceeds a certain rate, a 2-way interaction between protocol type and packet rate. Figure 1 illustrates how such a 2-way interaction may happen in code. Note that the failure will only be triggered when both pressure 300 are true.

Figure 1. 2-way interaction failure triggered only when two conditions are true.

But what if some failure is triggered only by a very unusual combination of 3, 4, or more sensor values? It is very unlikely that pairwise tests would detect this unusual case; we would need to test 3-way and 4-way combinations of values. But is testing all 4-way combinations enough to detect all errors? What degree of interaction occurs in real failures in real systems? Surprisingly, this question had not been studied when NIST began investigating interaction failures in 1999. Results showed that across a variety of domains, all failures could be triggered by a maximum of 4-way to 6-way interactions. As shown in Figure 2, the detection rate (y axis) increased rapidly with interaction strength (the interaction level t in t-way combinations is often referred to as strength). Studies by other researchers have been consistent with these results.

Figure 2. Error detection rates for interaction strengths 1 to 6

These results are interesting because they suggest that, while pairwise testing is not sufficient, the degree of interaction involved in failures is relatively low. Testing all 4-way to 6-way combinations may therefore provide reasonably high assurance. As with most issues in software, however, the situation is not that simple. Most parameters are continuous variables which have possible values in a very large range (+/- 232 or more). These values must be discretized to a few distinct values. In addition, we need to determine the correct result that should be expected from the system under test for each set of test inputs. But these challenges are common to all types of software testing, and a variety of good techniques have been developed for dealing with them.

Figure 3. 3-way covering array

A test suite to cover all 2-way, 3-way, etc. combinations is known as a covering array. An example is given in Figure 3, which shows a 3-way covering array for 10 variables with two values each. The interesting property of this array is that any three columns contain all eight possible values for three binary variables. For example, taking columns F, G, and H, we can see that all eight possible 3-way combinations (s000, 001, 010, 011, 100, 101, 110, 111) occur somewhere in the three columns together. In fact, any combination of three columns chosen in any order will also contain all eight possible values. Collectively, therefore, this set of tests will exercise all 3-way combinations of input values in only 13 tests, as compared with 1,024 for exhaustive coverage.

How can Combinatorial Methods be Used in Testing?

There are basically two approaches to combinatorial testing – use combinations of configuration parameter values, or combinations of input parameter values. In the first case, we select combinations of values of configurable parameters. For example, telecommunications software may be configured to work with different types of call (local, long distance, international), billing (caller, phone card, 800), access (ISDN, VOIP, PBX), and server for billing (Windows Server, Linux/MySQL, Oracle).

In the second approach, we select combinations of input data values, which then become part of complete test cases, creating a test suite for the application. For example, a word processing application may allow the user to select 10 ways to modify some highlighted text: subscript, superscript, underline, bold, italic, strikethrough, emboss, shadow, small caps, or all caps. Thorough testing requires that the font-processing function work correctly for all valid combinations of these input settings. But with 10 binary inputs, there are 210 = 1,024 possible combinations. But the empirical analysis reported above shows that testing all 3-way combinations may detect 90% or more of bugs. For a word processing application, testing that detects better than 90% of bugs may be a cost-effective choice. The covering array shown in Figure 3 could be used for this testing.

Conclusions

Using combinatorial methods for either configuration or input parameter testing can help make testing more effective at an overall lower cost. Although these methods do not apply to all applications, they can be a valuable addition to the tester’s toolbox.

Certain commercial products are identified in this document, but such identification does not imply recommendation by the US National Institute of Standards and Technology, nor does it imply that the products identified are necessarily the best available for the purpose.

(This article is condensed from Chapter 2 of Practical Combinatorial Testing, freely downloadable here)

Rick Kuhn is a computer scientist at the Computer Security Division of the US National Institute of Standards and Technology (NIST).
Raghu Kacker is a mathematical statistician at the Mathematical and Computational Sciences Division of NIST.
Yu Lei is an associate professor at the Department of Computer Science and Engineering at the University of Texas at Arlington.

Pairwise testing is one approach to solving the potential explosion in the number of tests when dealing with multiple parameters whose variables are semi-coupled or have some dependency on variable states of other parameters. For example, in the font dialog of MS Word there are 11 checkboxes for various effects such as superscript, strikethrough, emboss, etc. Obviously these effects have impact on how the characters in a particular font are displayed and can be used in multiple combinations such as Strikethrough + Subscript + Emboss. The total number of combinations of effects is the Cartesian product of the variables for each parameter, or 211 or 2048 in this example. This doesn’t include different font types, styles, etc. which also interdependent. So, you can see how the number of combinations increases rapidly especially as additional dependent parameters are included in the matrix.

The good news is the industry has a lot of evidence to suggest that most software defects occur from simple interactions between the variables of 2 parameters. So, from a risk based perspective where it may not be feasible to test all possible combinations how do we choose the combinations out of all the possibilities? Two common approaches include orthogonal arrays and combinatorial analysis.

But, true orthogonal arrays require that the number of variables is the same for all parameters. (Rarely true in software.) It is possible to create “mixed orthogonal arrays” where some combinations of variables will be tested more than once. For example, if we have 5 parameters and one parameter has 5 variables and the remains 4 parameters only have 3 variables each, we can see from the orthogonal array selector (available on FreeQuality website) the size of the orthogonal array is L25 (which basically means the test case will require 25 tests which is still significantly less than the total number of combinations of 405).

The other approach is combinatorial analysis (often referred to as pairwise or allpairs testing) because the approach most commonly used is to use a mathematical formula to reduce the total number of combinations in such a way that each variable for each parameter is tested with each variable from the other parameters at least once. In the above example, the number of tests would be reduced to 16. (Note: some tools will give slightly different results.) However, some tools (such as Microsoft’s PICT) also allow for more complex analysis of variable combinations such as triplets and n-wise coverage.

One problem that is hopefully not overlooked by testers using these tools is that some combinations of variables are simply not possible. For example, in the Effects group of the Font dialog it is impossible to check the Superscript checkbox and the Subscript checkbox simultaneously. Therefore, the tester either has to manually modify the output, or use a tool that allows constraints. Again, this is another situation where Microsoft’s free tool PICT excels. PICT uses a simply basic-like language for conditional and unconditional constraining of combinations of variables. PICT also allows weighting variables, seeding, output randomization, and negative testing.

I didn’t want this to be a PICT sales job, but alas my bias has influenced this post. So, I will conclude by pointing the readers to the Pairwise Testing website. My colleague Jacek Czerwonka has pulled together great resources on the technique of combinatorial analysis including a list of free and commercially available tools, and white papers supporting the value and practicality of this testing technique.

Article from Testing Mentor by BJ Rollison, Test Architect, Microsoft Inc

Developers of large data-intensive software often notice an interesting—though not surprising—phenomenon: When usage of an application jumps dramatically, components that have operated for months without trouble suddenly develop previously undetected errors. For example, newly added customers may have account records with an oddball combination of values that have not been seen before. Some of these rare combinations trigger faults that have escaped previous testing and extensive use. Alternatively, the application may have been installed on a different OS-hardware-DBMS-networking platform. Combinatorial testing can help detect problems like this early in the testing life cycle. The key insight underlying t-way combinatorial testing is that not every parameter contributes to every fault and many faults are caused by interactions between a relatively small number of parameters.

PAIRWISE TESTING

Suppose we want to demonstrate that a new software application works correctly on PCs that use the Windows or Linux operating systems, Intel or AMD processors, and the IPv4 or IPv6 protocols. This is a total of 2 × 2 × 2 = 8 possibilities but, as Table 1 shows, only four tests are required to test every component interacting with every other component at least once. In this most basic combinatorial method, known as pairwise testing, at least one of the four tests covers all possible pairs (t = 2) of values among the three parameters.

Note that while the set of four test cases tests for all pairs of possible values—for example, OS = Linux and protocol = IPv4—several combinations of three specific values are not tested—for example, OS = Windows, CPU = Intel, and protocol = IPv6.
Even though pairwise testing is not exhaustive, it is useful because it can check for simple, potentially problematic interactions with relatively few tests. The reduction in test set size from eight to four shown in Table 1 is not that impressive, but consider a larger example: a manufacturing automation system that has 20 controls, each with 10 possible settings—a total of 1020 combinations, which is far more than a software tester would be able to test in a lifetime. Surprisingly, we can check all pairs of these values with only 180 tests if they are carefully constructed.

Figure 1 shows the results of a 10-project empirical study conducted recently by Justin Hunter that compared the effectiveness of pairwise testing with manual test case selection methods.

The projects were conducted at six companies and tested commercial applications in development; in each project, two small teams of testers were asked to test the same application at the same time using different methods. One group of testers selected tests manually; they relied on “business as usual” methods such as developing tests based on functional and technical requirements and potential use cases mapped out on whiteboards. The other group used a combinatorial testing tool to identify pairwise tests. Test execution productivity was significantly higher in all of the projects for the testers using combinatorial methods, with test execution productivity more than doubling on average and more than tripling in three projects. The groups using pairwise testing also achieved the same or higher quality in all 10 projects; all of the defects identified by the teams using manual test case selection methods were identified by the teams using combinatorial methods. In five projects, the combinatorial teams found additional defects that had not been identifed by the teams using manual methods.

These proof-of-concept projects successfully demonstrated to the teams involved that manual methods of test case selection were not nearly as effective as pairwise combinatorial methods for finding the largest number of defects in the least amount of time.

TESTING HIGHER-DEGREE INTERACTIONS

Other empirical investigations have concluded that from 50 to 97 percent of software faults could be identified by pairwise combinatorial testing. However, what about the remaining faults? How many failures could be triggered only by an unusual interaction involving more than two parameters?

In a 1999 study of faults arising from rare conditions, the National Institute of Standards and Technology reviewed 15 years of medical device recall data to determine what types of testing could detect the reported faults (D.R. Wallace and D.R. Kuhn, “Failure Modes in Medical Device Software: An Analysis of 15 Years of Recall Data,” Int’l J. Reliability, Quality, and Safety Eng., Dec. 2001, pp. 351-371). The study found one case in which an error involved a four-way interaction among parameter values: demand dose = administered, days elapsed = 31, pump time = unchanged, and battery status = charged.

Pairwise combinatorial testing is unlikely to detect faults like this because it only guarantees that all pairs of parameter values will be tested. A particular four-way combination of values is statistically unlikely to occur in a test set that only ensures two-way combination coverage; to ensure thorough testing of complex applications, it is necessary to generate test suites for four-way or higher-degree interactions.

Investigations of other applications found similar distributions of fault-triggering conditions. Many faults were caused by a single parameter, a smaller proportion resulted from an interaction between two parameter values, and progressively fewer were triggered by three-, four-, five,- and six-way interactions.

Figure 2 summarizes these results. Thus far, a fault triggered by a seven-way interaction has not appeared.

With the Web server application, for example, roughly 40 percent of the failures were caused by a single value, such as a fie name exceeding a certain length; another 30 percent were triggered by the interaction of two parameters; and a cumulative total of almost 90 percent were triggered by three or fewer parameters.
While not conclusive, these results suggest that combinatorial methods can achieve a high level of thoroughness in software testing.

The key ingredient for this kind of testing is a covering array, a mathematical object that covers all t-way combinations of parameter values at least once. For the pairwise testing example in Table 1, t = 2, and it is relatively easy to generate tests that cover all pairs of parameter values. Generating covering arrays for complex interactions is much harder, but new algorithms make it possible to generate covering arrays orders of magnitude faster than previous algorithms, making up to six-way covering arrays tractable for many applications.

Figure 3 shows a covering array for all three-way interactions of 10 binary parameters in only 13 tests. Note that any three columns, selected in any order, contain all eight possible values of three parameters: 000,001,010,011, 100,101,110,111. Three-way interaction testing detected roughly 90 percent of bugs in all four of the empirical studies in Figure 2, but exhaustive testing of all possible combinations in Figure 3 would require 210 = 1,024 tests.

What are the pragmatic implications of being able to achieve 100 percent three-way coverage in 13 test cases on real-world software testing projects? Assuming that there are 10 defects in this hypothetical application and that 9 are identified through the 13 tests indicated, testing these 13 cases would find 71 times more
defects per test case [(9/13)/(10/1,024)] than testing exhaustively and uncovering all 10.

While the most basic form of combinatorial testing— pairwise—is well established, and adoption by software testing practitioners continues to increase, industry usage of these methods remains patchy at best. However, the additional training required is well worth the effort. Teams seeking to maximize testing thoroughness given tight time or resource constraints, and which currently rely on manual test case selection methods, should consider pairwise testing. When more time is available or more thorough testing is required, t-way testing for t > 2 is better. Practitioners who require very high quality software will find that covering arrays for higher-strength combinations can detect many hard to-find faults, and variability among detection rates appears to decrease as t increases.
Sophisticated new combinatorial testing algorithms packaged in user-friendly tools are now available to enable thorough testing with a manageable number of test cases and at lower cost, and make it practical for testers to develop empirical results on applications of this promising test method.

Article By

D. Richard Kuhn, Computer Scientist, NIST
Raghu Kacker, Mathematical Statistician, NIST
Yu Lei, Associate Professor, University of Texas at Arlington
Justin Hunter, CEO, Hexawise

Download here

Dr. Cem Kaner – Director, Center for Software Testing Education & Research, Florida Institute of Technology

PC World Vietnam: What did you think of VISTACON 2010?

Dr. Kaner: I am very impressed that the event was very professionally organized and happy to meet my old colleagues to share and exchange more about our area of expertise. I was also very excited being one of the speakers at the conference and presented on some topics which focused on software testing as well as sharing some of my experiences from my research and work in this field.

PC World Vietnam: What do you believe are the most important skills required for a tester?

Dr. Kaner: Testers should have a good knowledge of the profession that they are working in; have testing skills and certain technology basics. For example, when testing banking or finance software, you should know about their trading activities, how they liquidate, and have a thorough grasp of some of the testing tools, testing plans and test automation methods. The technology basics means you should grasp some steps of the software development process, programming language skills, a knowledge of algorithms and operating systems. Besides that, other essential factors that can help you become a successful tester are having a good observant mind to easily recognize bugs during the testing process, the passion to apply the highest standards to the project, and finally, that you are always interested in new technology and new products.

PC World Vietnam: Could you please share with us your experience of undertaking testing projects in different countries?

Dr. Kaner: I have had the opportunity to work and collaborate on projects in the US and India in the field of software testing. From my experience, you should focus on the test cases created during the period you are creating code. Parallel to that, you need to improve the testing process to achieve the desired quality.

It is important to find ways to solve problems by preventing bugs instead of finding bugs. Testers should talk and share their experiences of finding bugs to improve their professional skills. Also, the wages paid to a tester in each country varies. For instance, in the US, the cost of hiring a tester is very high, so most companies outsource their projects to another country to save costs. According to Global Services Media, last year Ho Chi Minh City ranked 5th in the list top 50 emerging global cities in outsourcing software. That is a good sign for the development of the Vietnamese software industry.

PC World Vietnam: Vietnamese students often tend to become programmers instead of testers, what do you think about this?

Dr. Kaner: Actually, the hobbies and passions of each person will always be different. I myself have been rotated between the two positions of testing team manager and software programmers’ team manager. Some people find the testing field interesting, but others discover their own skills are more in line with programming. Those who prefer to focus on solving a problem or specific solutions may be appropriate as a programmer. But if you decide you want to become a tester, you should do your best to make this happen by studying hard and doing serious research.

PC World Vietnam: Thank you, Dr. Kaner

(This interview is condensed and translated from the PC World Vietnam Magazine – October 2010 Issue)