Internet-based per-use service models are turning things upside down in the software development industry, prompting rapid expansion in the development of some products and measurable reduction in others. (Gartner, August 2008) This global transition toward computing “in the Cloud” introduces a whole new level of challenge when it comes to software testing.

Cloud computing, when it is done well, provides a reliable and single point of access for users. Consistent, positive user experience sells the service, and rigorous testing assures a quality experience. In order to produce reliable, effective results for users of many walks of life, exacting software testing standards must be met.

In a series of articles, LogiGear is identifying four fundamental requirements by which software testing in the Cloud can uphold these standards:

Four Fundamental Requirements for Successful Testing in the Cloud:

1. Constantly Test Everything – the Right Way
2. Know What’s Where – and Prove It
3. Define Your Paradigm
4. Don’t Underestimate the Importance of Positive User Experience

In this issue’s article, we address:

Requirement 03: Define your Paradigm

Today’s companies increasingly find that they are in an ever more competitive market, especially in the drive to implement more robust, capable and pioneering Cloud-based products and services. Product delivery times are decreasing, customers demand higher and higher levels of product quality, and failure to deliver within the customer’s expectations can be swiftly punished with whole scale product abandonment and erection of barriers to market reentry.

Companies who are leading creative efforts to address these volatile areas recognize that quality must be uncompromised, and indeed surpassed in every way.

Adopting a testing paradigm that is designed specifically for the requirements of Cloud-computing is a fundamental requirement for the new standards of quality being set by customer-driven demand.

The steps for defining a Cloud-friendly testing paradigm are in general:

1. Evaluate Your Current Testing Paradigm
2. Define Paradigm Requirements that Meet Standards for Cloud-based Technologies
3. Apply Proven Methods and Technology

1. Evaluate Your Current Testing Paradigm

Each quality or development team has its own frame of reference by which it identifies the need for testing overall, and the specific kinds of testing required. This frame of reference, or paradigm, often reflects the higher organizational approach to quality assurance and the role that testing is perceived to occupy in customer satisfaction and loyalty.

Cloud computing introduces a need for a new paradigm – one that incorporates the 24/7 testing-on-the-fly required for successful Software as a Service implementation and other Cloud-based products.

Some organizations may find their current approach to testing adapts relatively well for the “test everything all the time” model. However, most will find it necessary to reassess not just their approach toward testing, but the fundamental principles that underlie that approach – ones they’ve inherited from traditional testing models.

For a more detailed look at the historical development of testing methodologies, read our white paper “The 5% Solution”

In the world of software testing, all the players used to start in pretty much the same place: manual testing. As the industry matured, more elegant testing solutions – record and playback, automated or scripted testing, and action-based testing – emerged over time. Each new approach to testing successively increased our testing efficiencies, reporting efficiencies and ability to handle larger and larger test loads, lending leverage to the leading few. Conducting a Current-State Assessment can give you the necessary insight into what works and doesn’t work in your current testing paradigm when it comes to testing in the Cloud.

Conducting a Current-State Assessment

Most companies have tried a variety of testing approaches, some manual, some scripted, and maybe even some attempts at automation. It is important for a firm to conduct an honest self-assessment about their current testing footprint, capabilities, limitations, and opportunities for improvement.

Many times firm have a belief about their current testing approach and are surprised to find that those beliefs, albeit hopeful, are not fully rooted in practice. The following questions will help thoroughly evaluate your testing paradigm.

• Testing methodology – Is it largely manual or some combination of methods? Are those methods consistent?
• Testing environment – Where are the applications located? Where is the data located? What are the network and performance constructs?
• People resources and skill sets – What is the competency makeup of your current test team? What are the skill sets required for moving forward?
• Time-to-deliver constraints – What are the barriers to testing efficiency? What are the efforts that take more time than others and why?
• Reporting and progress visibility – Do you have consistent visibility into your testing status? Are you surprised when testing efforts are late? Are statuses verbal, or based on actual metrics?

2. Define Paradigm Requirements that Meet Standards for Cloud-based Technologies

Achieving your organization’s testing goals, in whole or in part, requires planning and mapping out your “testing in the Cloud” strategy and its requirements.

Clarifying Cloud Architecture

An initial understanding of the different types of clouds and what their high-level architectures offer will inform the kind of testing paradigm you will need to establish.

Sam Johnston’s integrated picture depicts the three cloud types:

• Remote, Public or External Cloud – Public clouds are sometimes referred to as “Regular” cloud computing. Completely separate from a user’s desktop or the corporate network that they belong to, public clouds offer a pay-per-use service model because the user is leveraging outside compute resources for the particular service they are seeking.This approach offers economies of scale, but their shared infrastructure model can raise concerns about configuration restrictions, adequate security, and service-levels (available up-time). These concerns might make you think twice about subjecting sensitive data that is subject to compliance or safe harbor regulations.Because “public” clouds are typically made available via the public internet they may be free or inexpensive to use. A well-known example of a public cloud is Amazon EC2 which is available for use by the general public.
• Internal or Private Cloud – Private cloud computing extends the same infrastructure concepts firms already have in their data centers. The motivation for private clouds appears to be to resolve security and availability concerns inherent in the public cloud paradigm.As such, private clouds seemingly are not burdened by network bandwidth, availability issues or potential security risks that may be associated with public clouds. However, this thinking belies the very intent of cloud computing which is predicated on hardware-software extensibility, dramatic reduction in infrastructure costs, and an elimination of the management concerns governing private networks.
• Mixed or Hybrid Cloud – Many of the leading engineering thinkers in the industry suggest that the most workable cloud computing approach is the “hybrid” approach. The hybrid solution combines the best of the Public and Private Cloud paradigms.Considering that some applications may be too complex or too sensitive to risk operating from a public cloud it makes sense for a firm to protect those application and data assets within the construct of a private cloud where they have total control.Less sensitive applications and data can be migrated to a public cloud freeing compute resources that can be re-purposed for the complex applications that need to stay home.The hybrid approach does sound like the best of both worlds. It makes sense from a technology and economic standpoint. It allows for control, flexibility and growth. The trick to managing hybrid clouds comes when you consider spikes in demand.When demand spikes pummel the performance of your applications located within the private cloud -and you need additional computing power (such as is experienced by web-based news media when critical events occur) -you will need to develop a management policy that can be responsible for when to reach out to the public cloud for those additional resources.

Defining the Future-State

Envisioning the results of your testing transformation requires solid understanding of your organization’s business goals and objectives, the Cloud computing paradigms that may help your testing effort contribute to those goals, and development of a sound plan to move in the new direction.

When documenting your planned future state, address each of these categories:

• Architectural – Consider the different Cloud paradigms as they pertain to your business model, goals and objectives, and application and data sensitivity.
• Organization – Ensure organizational review and consensus of new Cloud testing direction as it pertains to business goals and objectives and priorities.
• Financial – Define the benefits, know where the real costs lie, and define the budget.
• Implementation – Adopt an incremental improvement approach, and choose the correct tools and partners.
• Monitor and measure – Develop a consistent set of metrics for measuring and monitoring your new foray into testing in the Cloud.

Other organizational goals and objectives that are important to include:

• Provide sufficient support for distributed test teams
• Increase your return-on-testing-investment
• Significantly decrease time-to-market
• Optimize the reusability of tests and test automation
• Improve test output and coverage
• Enhance the motivation of your testing staff
• Increase managerial control over quality and testing
• Have better visibility into quality and testing effectiveness

While setting your testing goals, consider that in the LogiGear white paper “The 5% Solution,” Hans Buwalda, LogiGear’s CTO, posits that “with good test automation you can have more tests and execute them any time you need to, thus significantly improving the development cycles of your project.”

With the repeatable tests under automation, he says, testers are free to work on more sophisticated testing as well as testing focused on new features and functions increasing the overall quality of the delivered result.

Buwalda establishes the 5% Goals:

• No more than 5% of structured test cases should be tested manually
• No more than 5% of testing efforts should be spent in the automation process

Conversely, the five percent goals suggest that 95% of your tests should be automated and 95% of your testing efforts should be spent on emerging functionality and higher complexity testing. Other organizational goals fall into some pretty predictable categories.

3. Apply Proven Methods and Technology

Implementing the changes necessary to adopt a new testing paradigm will bear out how well you’ve defined your requirements. Therefore, selecting well-matched testing tools and specific architectural approaches can have a dramatic impact on the results of your paradigm shift.

Architectural Approaches

If you are going to perform testing in the cloud it is important to understand the different architectural approaches that are available and the unique advantages that each approach offers. First, let’s start out with a quick list of the six basic cloud architectures or relationships.

Data storage, duplication and backup

Cloud Accessible Application

This category of cloud computing technology focuses on applications that are entirely externally hosted at a cloud services application vendor. No installation of software is required on the user desktop. The application functionality is utilized through a web browser and the data is stored on the application host’s servers. Salesforce.com is a good example of this type of application service.

Cloud-based Client Services

Think of a device that is completely useless without a connection to internet services (like the iPhone) and you will have an understanding of what cloud-based client services are all about. Netbooks are another emerging computing approach predicated on utilizing more cloud-based applications and services using a smaller, less powerful laptop computer.

Scalable Infrastructure and Computational Arrays

Imagine a sudden spike in demand for products you want to sell. Good news, right? But let’s image that your computer room server just can’t handle the additional load. You lose money with every customer who abandons their purchase. With the right cloud infrastructure provider relationship those worries might be a thing of the past. Vendors are increasing building expanding infrastructure architectural offers to support this very problem. Good examples to keep in mind are Amazon and Sun Microsystems.

Application Development and Deployment Platforms

Picture the scenario that you have developed a ‘killer’ application that you know everyone is going to want but you can’t deal with the cost and complexity of buying and managing the server and network equipment to manage it. Web hosting services are a good example of this approach allowing individuals and organizations to provide their own web site accessible via the web. Still other platforms allow for remote applications development and deployment.

Utility Services

MapQuest is popular example of a cloud-based service utility. It is completely inaccessible and unusable if you are not connected to the web. Other examples include, PayPal and Yahoo search.

Data Storage, Duplication and Backup

There are a number of variations of data services available via the cloud. Simple data storage allows for a pay-as-you-increase sort of service allowing ever increasing disk utilization as needed. Other services offer data duplication and mirroring, and data backup.

Conclusion

In this paper, we’ve presented the need to “define your paradigm” as a necessary prerequisite for pursuing a strategy for “testing-in-the-cloud”. We began with the common-sense directive of evaluating your current testing approach and setup. We then reviewed the different cloud types, their focus, and their typical use. Additionally, we offered the business objectives most firms would consider as foundation for making a move to a Cloud-based testing paradigm. Lastly, we presented the six common cloud computing paradigms, the industry standards that have emerged to support each, and introduced examples of services in each category.

Your testing-in-the-Cloud strategy should consider all of these approaches to determine what will work best for your organization. Your success is will be determined by a disciplined approach and thorough implementation. Happy testing!