Internet-based per-use service models are turning things upside down in the software development industry, prompting rapid expansion in the development of some products and measurable reduction in others. (Gartner, August 2008) This global transition toward computing “in the Cloud” introduces a whole new level of challenge when it comes to software testing.

Cloud computing, when it is done well, provides a reliable and single point of access for users. Consistent, positive user experience sells the service, and rigorous testing assures a quality experience. In order to produce reliable, effective results for users of many walks of life, exacting software testing standards must be met.

In a series of articles, LogiGear is identifying four fundamental requirements by which software testing in the Cloud can uphold these standards:

Four Fundamental Requirements for Successful Testing in the Cloud:

01:Constantly Test Everything – the Right Way

02:Know What’s Where – and Prove It

03:Define Your Paradigm

04:Don’t Underestimate the Importance of Positive User Experience

In this issue’s article, we address:

Requirement 02: Know What’s Where – and Prove It

Although some would argue to the contrary, the nature of Cloud computing creates inherent risks not encountered in traditional software development. Financial transactions and the storing of Protected Consumer Information, especially on Internet-connected systems, open the door to potential data losses, undesirable user experiences… and audit compliance issues. Responsibilities once relegated to local IT departments are becoming core competencies for QA organizations. Many companies are finding that it makes sense to bring in an experienced Cloud testing partner to help the in-house team avoid these problems.

Introducing relationships with third-party partners, Cloud-based datacenters and application hosting services adds an additional level of complexity to tracking and demonstrating data security.

Interruptions in Cloud-based services are troubling because companies are increasingly evolving their operations to run critical software as well as storing larger and larger amounts of data in the Cloud computing paradigm. So if service is interrupted, organizations may find that they are unable to function.

For our discussion of the right tools and methodology, read the first installment of this series: Requirement 01: Constantly Test Everything – the Right Way.

There are several approaches major firms take to better insure the utility of the Cloud services they receive.

1. Insist on service-level agreements – Google offers service level agreements (SLA) promising that components of their offerings will be available as much as 99.9 percent of the time (9 hours per year down time), with service credits if availability drops below the agreed upon levels. The only way companies can realistically achieve this model is to have superior test sets, that are fully automated – and to go about automation the right way. Otherwise it can quickly become unachievable and unmanageable.
2. Transparency through dashboards – Make use of the Cloud vendor’s dashboards and service interruption alerts. Google, Yahoo, Amazon and other major players give developed administrative dashboards tracking status availabilities, downtimes, notifications and alerts. Salesforce.com shows the response times for their Salesforce.com server transactions. They detail problems, their affects on system availability and other related applications, and, more importantly what caused the problems and what the mitigations are.
3. Cloud monitoring services – This emerging sector of the Cloud services spectrum dives deeper into the details of Cloud availability, status, service interruptions and problem mitigations.
4. Automate Cloud testing – As you integrate a variety of applications on the Cloud it’s important to be able to quickly test your work flows when introducing a new or revised app.

In addition to these approaches, asking four key questions can assist in mitigating some of the potential outsourcing risks and help drive a more secure and predictable partnership:

1. Know Yourself
2. Know Your Partner
3. Determine Your Model
4. Audit Activity & Remediate Issues

Step 1: Know Yourself

Detailed understanding of your application, data usage, security and risk profile is necessary when preparing to engage a third-party testing partner. Here are some tips to take into consideration as you assemble a list of requirements:

Application Architecture:

• Performance – Testing a complex Cloud application can be complicated by the response latency of sites containing huge objects. Be sure to anticipate what any performance bottleneck might be with your outsourced testing partner, as even small delays can have significant impact on the efficiency of their testing.
• Direct connect – It is important to be aware of all of the components that make up the connection between you, your outsourced testing partner and your Cloud-based service provider – not just for efficiency’s sake, but to make sure that your data doesn’t take any unwanted detours between here and there.

A broad-reaching 12-month survey by Orthos Corp found that “…every organization without exception had suffered multiple instances of data leakage, many of them serious and potentially damaging.”

Data Architecture:

• Data in transit – Encrypting data in transit is far more challenging. Be sure to identify or implement specific products and strategies to encrypt data so that it cannot be read or tampered with as it crosses the networks between you and your outsourced testing partner.

Systems Architecture:

• Map out the route – Document each stopping and transition point on the route data takes across networks during testing activity. Ensure that data are protected from the beginning of the journey to the journey’s end. Constructing an effective strategy to protect your data involves knowing where that data travels to and what happens to it along the way.
• Evaluate efficiency – Ensure that your Cloud partner is not paying a prohibitive performance penalty due to accessing, unencrypting or otherwise unwinding the security protection wrapping your data. Take into consideration available and accessible network bandwidths (effective end-to-end bandwidth), and any data or application and query transaction latencies.
• Audit for leakage – Look beyond available bandwidth to see what the effective capabilities are with regard to throughput, latency and redundancy. There are a large number of vendors who will conduct data transmission and leakage audits, providing a complete end-to-end data traceability, content verification and inspection.

Step 2: Know Your Partner

Trust but verify.

Your relationship with an outsourced testing partner is by nature an intimate one – exposing your data, business practices, clients and customers, process flows and service offerings. Your relationship with this important partner must be well-defined and predicated on trust, especially when working together in the Cloud.

Tips for defining your relationship with your third-party testing partner:

• Scope it out – Carefully document your expectations regarding use and limitations of data, data storage, proliferations, etc. Define the compliance expectations, audit approaches, and remediation baselines and thresholds.
• Lay on the legalese – Obtain a signed non-disclosure confidentiality agreement and other protective legal documentation.
• Assess capabilities – Validate your testing partner’s capabilities both from a testing capacity as well as a security compliance capacity. Are they located at a secure facility? Do they have internal governance policies that dictate their focus on protecting customer data? Are their computer and network systems capable of providing the encryption and data storage functions needed?
• Select bonded services – Require that your testing partner is bonded and that they bond their employees. A fidelity ‘Bond’ is usually a guarantee against dishonesty such as data and identity theft. Most fidelity bonds have an arrest and conviction clause and often prove to be a powerful deterrent.

Step 3: Determine Your Model

It is important to map out where your data assets are located, how they interrelate, and how they should be segregated.

• Role-based systems – Document not only details about the data but details about who uses the data and why. Roles within a process should be allowed only specific uses of data related to defined tasks.
• Data access model – Firms with lasting outsourcing relationships with testing partners carefully create a data usability map to:
• Identify users and assign roles / access permissions.
• Decide architecturally where user authentication will occur. (e.g. user IDs and all attributes known only to the Web server).
• Define session constraints, Single Instance rules, data volume limits, etc.
• Decide monitoring functions, alerts, and reports.

Step 4: Audit Activity & Remediate Issues

• Validate that things are running smoothly on a periodic basis:
• Collect clickstream information.
• Determine what level of granularity you want to track and how often (periodicity).
• Track the types and volumes of testing activity (manual testing vs. automated testing).
• Report on the activity.
• Identify the types of users and match names to user accounts and provide reporting of access and usage.
• Conduct other ongoing protective oversight, monitoring, logging and reporting.
• Track and report on system availability.
• Remediate performance and security issues based on expectations (baselines) and boundaries (thresholds) determined in Step 2.

When measuring the success of your partnership with an outsourced testing partner, include these steps as part of a review checklist. Documentation, diligence and transparency from both parties will go a long way to keeping that Cloud locked down tight.