Four Fundamental Requirements of Successful Testing in the Cloud – Part II

Internet-based per-use service models are turning things upside down in the software development industry, prompting rapid expansion in the development of some products and measurable reduction in others. (Gartner, August 2008) This global transition toward computing “in the Cloud” introduces a whole new level of challenge when it comes to software testing.

Cloud computing, when it is done well, provides a reliable and single point of access for users. Consistent, positive user experience sells the service, and rigorous testing assures a quality experience. In order to produce reliable, effective results for users of many walks of life, exacting software testing standards must be met.

In a series of articles, LogiGear is identifying four fundamental requirements by which software testing in the Cloud can uphold these standards:

Four Fundamental Requirements for Successful Testing in the Cloud:

01:Constantly Test Everything – the Right Way
02:Know What’s Where – and Prove It
03:Define Your Paradigm
04:Don’t Underestimate the Importance of Positive User Experience

In this issue’s article, we address:

Requirement 02: Know What’s Where – and Prove It

Although some would argue to the contrary, the nature of Cloud computing creates inherent risks not encountered in traditional software development. Financial transactions and the storing of Protected Consumer Information, especially on Internet-connected systems, open the door to potential data losses, undesirable user experiences… and audit compliance issues. Responsibilities once relegated to local IT departments are becoming core competencies for QA organizations. Many companies are finding that it makes sense to bring in an experienced Cloud testing partner to help the in-house team avoid these problems.

Introducing relationships with third-party partners, Cloud-based datacenters and application hosting services adds an additional level of complexity to tracking and demonstrating data security.

Interruptions in Cloud-based services are troubling because companies are increasingly evolving their operations to run critical software as well as storing larger and larger amounts of data in the Cloud computing paradigm. So if service is interrupted, organizations may find that they are unable to function.

For our discussion of the right tools and methodology, read the first installment of this series: Requirement 01: Constantly Test Everything – the Right Way.

There are several approaches major firms take to better insure the utility of the Cloud services they receive.

  1. Insist on service-level agreements – Google offers service level agreements (SLA) promising that components of their offerings will be available as much as 99.9 percent of the time (9 hours per year down time), with service credits if availability drops below the agreed upon levels. The only way companies can realistically achieve this model is to have superior test sets, that are fully automated – and to go about automation the right way. Otherwise it can quickly become unachievable and unmanageable.
  2. Transparency through dashboards – Make use of the Cloud vendor’s dashboards and service interruption alerts. Google, Yahoo, Amazon and other major players give developed administrative dashboards tracking status availabilities, downtimes, notifications and alerts. shows the response times for their server transactions. They detail problems, their affects on system availability and other related applications, and, more importantly what caused the problems and what the mitigations are.
  3. Cloud monitoring services – This emerging sector of the Cloud services spectrum dives deeper into the details of Cloud availability, status, service interruptions and problem mitigations.
  4. Automate Cloud testing – As you integrate a variety of applications on the Cloud it’s important to be able to quickly test your work flows when introducing a new or revised app.

In addition to these approaches, asking four key questions can assist in mitigating some of the potential outsourcing risks and help drive a more secure and predictable partnership:

  1. Know Yourself
  2. Know Your Partner
  3. Determine Your Model
  4. Audit Activity & Remediate Issues

Step 1: Know Yourself

Detailed understanding of your application, data usage, security and risk profile is necessary when preparing to engage a third-party testing partner. Here are some tips to take into consideration as you assemble a list of requirements:

Application Architecture:

  • Performance – Testing a complex Cloud application can be complicated by the response latency of sites containing huge objects. Be sure to anticipate what any performance bottleneck might be with your outsourced testing partner, as even small delays can have significant impact on the efficiency of their testing.
  • Direct connect – It is important to be aware of all of the components that make up the connection between you, your outsourced testing partner and your Cloud-based service provider – not just for efficiency’s sake, but to make sure that your data doesn’t take any unwanted detours between here and there.
A broad-reaching 12-month survey by Orthos Corp found that “…every organization without exception had suffered multiple instances of data leakage, many of them serious and potentially damaging.”

Data Architecture:

  • Data in transit – Encrypting data in transit is far more challenging. Be sure to identify or implement specific products and strategies to encrypt data so that it cannot be read or tampered with as it crosses the networks between you and your outsourced testing partner.

Systems Architecture:

  • Map out the route – Document each stopping and transition point on the route data takes across networks during testing activity. Ensure that data are protected from the beginning of the journey to the journey’s end. Constructing an effective strategy to protect your data involves knowing where that data travels to and what happens to it along the way.
  • Evaluate efficiency – Ensure that your Cloud partner is not paying a prohibitive performance penalty due to accessing, unencrypting or otherwise unwinding the security protection wrapping your data. Take into consideration available and accessible network bandwidths (effective end-to-end bandwidth), and any data or application and query transaction latencies.
  • Audit for leakage – Look beyond available bandwidth to see what the effective capabilities are with regard to throughput, latency and redundancy. There are a large number of vendors who will conduct data transmission and leakage audits, providing a complete end-to-end data traceability, content verification and inspection.

Step 2: Know Your Partner

Trust but verify.

Your relationship with an outsourced testing partner is by nature an intimate one – exposing your data, business practices, clients and customers, process flows and service offerings. Your relationship with this important partner must be well-defined and predicated on trust, especially when working together in the Cloud.

Tips for defining your relationship with your third-party testing partner:

  • Scope it out – Carefully document your expectations regarding use and limitations of data, data storage, proliferations, etc. Define the compliance expectations, audit approaches, and remediation baselines and thresholds.
  • Lay on the legalese – Obtain a signed non-disclosure confidentiality agreement and other protective legal documentation.
  • Assess capabilities – Validate your testing partner’s capabilities both from a testing capacity as well as a security compliance capacity. Are they located at a secure facility? Do they have internal governance policies that dictate their focus on protecting customer data? Are their computer and network systems capable of providing the encryption and data storage functions needed?
  • Select bonded services – Require that your testing partner is bonded and that they bond their employees. A fidelity ‘Bond’ is usually a guarantee against dishonesty such as data and identity theft. Most fidelity bonds have an arrest and conviction clause and often prove to be a powerful deterrent.

Step 3: Determine Your Model

It is important to map out where your data assets are located, how they interrelate, and how they should be segregated.

  • Role-based systems – Document not only details about the data but details about who uses the data and why. Roles within a process should be allowed only specific uses of data related to defined tasks.
  • Data access model – Firms with lasting outsourcing relationships with testing partners carefully create a data usability map to:
    • Identify users and assign roles / access permissions.
    • Decide architecturally where user authentication will occur. (e.g. user IDs and all attributes known only to the Web server).
    • Define session constraints, Single Instance rules, data volume limits, etc.
    • Decide monitoring functions, alerts, and reports.

Step 4: Audit Activity & Remediate Issues

  • Validate that things are running smoothly on a periodic basis:
    • Collect clickstream information.
    • Determine what level of granularity you want to track and how often (periodicity).
    • Track the types and volumes of testing activity (manual testing vs. automated testing).
    • Report on the activity.
    • Identify the types of users and match names to user accounts and provide reporting of access and usage.
    • Conduct other ongoing protective oversight, monitoring, logging and reporting.
    • Track and report on system availability.
  • Remediate performance and security issues based on expectations (baselines) and boundaries (thresholds) determined in Step 2.

When measuring the success of your partnership with an outsourced testing partner, include these steps as part of a review checklist. Documentation, diligence and transparency from both parties will go a long way to keeping that Cloud locked down tight.

LogiGear Corporation

LogiGear Corporation provides global solutions for software testing, and offers public and corporate software-testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast and cost-effective results. Since 1994, LogiGear has worked with hundreds of companies from the Fortune 500 to early-stage startups, creating unique solutions to exactly meet their needs. With facilities in the US and Vietnam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

For more information, contact Joe Hughes + 01 650.572.1400

LogiGear Corporation
LogiGear Corporation provides global solutions for software testing, and offers public and corporate software testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast, cost-effective results. Since 1994, LogiGear has worked with Fortune 500 companies to early-stage start-ups in, creating unique solutions to meet their clients’ needs. With facilities in the US and Viet Nam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

The Related Post

Introduction This article provides a business case for considering Vietnam as an offshore outsourcing destination. It starts by outlining a framework for evaluating potential outsourcing destinations. It then proceeds to evaluate Vietnam against the framework, demonstrating that Vietnam is worthy of strong consideration as an offshore outsourcing destination.
Options for getting outsourced testing help when you need it most. There are times when organizations need to increase testing for a regular release, to meet a deadline or some unexpected occurrence, but it doesn’t justify expanding the inside team. The options are to quickly find someone for a short term contract, or depending on ...
Rain is a constant in Vietnam. While sometimes it may inconvenience us, it is one of the country’s most valuable natural resources. Rain is one of Vietnam’s defining characteristics. It floods the streets, feeds the fields, cleans the cities and provides a soothing soundtrack at night. Some days we curse it, some days we embrace ...
This article was adapted from a presentation titled “Build a Successful Global Training Program” to be presented by Michael Hackett, LogiGear Vice President, Business Strategy and Operations, at the Software Test & Performance Conference 2006 at the Hyatt Regency Cambridge, Massachusetts (November 7 – 9, 2006). Introduction This article describes how best to prepare yourself ...
A lot of people still think primarily about cost-cutting as the main motivation to outsource software development projects to offshore or nearshore providers. However, in recent years the industry has greatly evolved and many providers have greatly invested in the quality of their services.
Da Nang City, March 12, 2011 – LogiGear Corporation will officially open its Software Testing and Research Center in the Dana Book Building at 76-78 Bach Dang Street, Hai Chau District, Da Nang City. The opening ceremony will welcome representatives from Da Nang and Duy Tan University as well as government agencies.
Tết holidays is a celebration of family, food and a favorable new year. February third celebrates Tết Nguyên Đán, or otherwise known as Tết, the Lunar New Year holiday welcoming the year of the cat. The holiday varies from late January to early February officially lasting three days. More recently, China and Vietnam celebrate Tết ...
An Interview With Hung Nguyen, Founder and CEO of LogiGear Corporation This interview was published by Saigon Entrepreneur Weekly (Issue 186, February 2, 2007), the most respected and widely read business magazine in Vietnam. Hung Nguyen is a well known author of best selling books about software testing in the United States and is a ...
Description: This two-day course focuses on developing a strategic approach to distributed test project management, effective communication, bug-database management and metrics, resource evaluation, and successful test execution, as well as many of the “soft” skills needed to lead and manage offshore teams. Implementation and use of test management tools and documentation will also be covered. ...
Today, many test team leaders must continue to successfully ensure the quality of their applications under test (AUT) while dealing with the increased challenge of having some or even all of their team members located offshore. This article, part 1 of a 2-part discussion, will take a look at the unique risks faced by a ...
Vietnam is a country of contrasts due to its geography. This makes the country unique as it offers travelers the ability to experience a multitude of landscapes. From the rice-cultivating tropical lowlands, to the rich-soiled, coffee-producing highlands, Vietnam offers a unique and varied travel experience. 22% of Vietnam’s 87 million people live in the Mekong ...
BJ Rollison, Software Test Architect for Microsoft, is planning to speak at the Viet Nam International Software Testing Automation Conference 2010 (VISTACON 2010) hosted by LogiGear Corporation from 20 – 22 September 2010. Mr. Rollison started working for Microsoft in 1994, becoming one of the leading experts of test architecture and execution at Microsoft. He ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news