Four Fundamental Requirements of Successful Testing in the Cloud – Part II

Internet-based per-use service models are turning things upside down in the software development industry, prompting rapid expansion in the development of some products and measurable reduction in others. (Gartner, August 2008) This global transition toward computing “in the Cloud” introduces a whole new level of challenge when it comes to software testing.

Cloud computing, when it is done well, provides a reliable and single point of access for users. Consistent, positive user experience sells the service, and rigorous testing assures a quality experience. In order to produce reliable, effective results for users of many walks of life, exacting software testing standards must be met.

In a series of articles, LogiGear is identifying four fundamental requirements by which software testing in the Cloud can uphold these standards:

Four Fundamental Requirements for Successful Testing in the Cloud:

01:Constantly Test Everything – the Right Way
02:Know What’s Where – and Prove It
03:Define Your Paradigm
04:Don’t Underestimate the Importance of Positive User Experience

In this issue’s article, we address:

Requirement 02: Know What’s Where – and Prove It

Although some would argue to the contrary, the nature of Cloud computing creates inherent risks not encountered in traditional software development. Financial transactions and the storing of Protected Consumer Information, especially on Internet-connected systems, open the door to potential data losses, undesirable user experiences… and audit compliance issues. Responsibilities once relegated to local IT departments are becoming core competencies for QA organizations. Many companies are finding that it makes sense to bring in an experienced Cloud testing partner to help the in-house team avoid these problems.

Introducing relationships with third-party partners, Cloud-based datacenters and application hosting services adds an additional level of complexity to tracking and demonstrating data security.

Interruptions in Cloud-based services are troubling because companies are increasingly evolving their operations to run critical software as well as storing larger and larger amounts of data in the Cloud computing paradigm. So if service is interrupted, organizations may find that they are unable to function.

For our discussion of the right tools and methodology, read the first installment of this series: Requirement 01: Constantly Test Everything – the Right Way.

There are several approaches major firms take to better insure the utility of the Cloud services they receive.

  1. Insist on service-level agreements – Google offers service level agreements (SLA) promising that components of their offerings will be available as much as 99.9 percent of the time (9 hours per year down time), with service credits if availability drops below the agreed upon levels. The only way companies can realistically achieve this model is to have superior test sets, that are fully automated – and to go about automation the right way. Otherwise it can quickly become unachievable and unmanageable.
  2. Transparency through dashboards – Make use of the Cloud vendor’s dashboards and service interruption alerts. Google, Yahoo, Amazon and other major players give developed administrative dashboards tracking status availabilities, downtimes, notifications and alerts. shows the response times for their server transactions. They detail problems, their affects on system availability and other related applications, and, more importantly what caused the problems and what the mitigations are.
  3. Cloud monitoring services – This emerging sector of the Cloud services spectrum dives deeper into the details of Cloud availability, status, service interruptions and problem mitigations.
  4. Automate Cloud testing – As you integrate a variety of applications on the Cloud it’s important to be able to quickly test your work flows when introducing a new or revised app.

In addition to these approaches, asking four key questions can assist in mitigating some of the potential outsourcing risks and help drive a more secure and predictable partnership:

  1. Know Yourself
  2. Know Your Partner
  3. Determine Your Model
  4. Audit Activity & Remediate Issues

Step 1: Know Yourself

Detailed understanding of your application, data usage, security and risk profile is necessary when preparing to engage a third-party testing partner. Here are some tips to take into consideration as you assemble a list of requirements:

Application Architecture:

  • Performance – Testing a complex Cloud application can be complicated by the response latency of sites containing huge objects. Be sure to anticipate what any performance bottleneck might be with your outsourced testing partner, as even small delays can have significant impact on the efficiency of their testing.
  • Direct connect – It is important to be aware of all of the components that make up the connection between you, your outsourced testing partner and your Cloud-based service provider – not just for efficiency’s sake, but to make sure that your data doesn’t take any unwanted detours between here and there.
A broad-reaching 12-month survey by Orthos Corp found that “…every organization without exception had suffered multiple instances of data leakage, many of them serious and potentially damaging.”

Data Architecture:

  • Data in transit – Encrypting data in transit is far more challenging. Be sure to identify or implement specific products and strategies to encrypt data so that it cannot be read or tampered with as it crosses the networks between you and your outsourced testing partner.

Systems Architecture:

  • Map out the route – Document each stopping and transition point on the route data takes across networks during testing activity. Ensure that data are protected from the beginning of the journey to the journey’s end. Constructing an effective strategy to protect your data involves knowing where that data travels to and what happens to it along the way.
  • Evaluate efficiency – Ensure that your Cloud partner is not paying a prohibitive performance penalty due to accessing, unencrypting or otherwise unwinding the security protection wrapping your data. Take into consideration available and accessible network bandwidths (effective end-to-end bandwidth), and any data or application and query transaction latencies.
  • Audit for leakage – Look beyond available bandwidth to see what the effective capabilities are with regard to throughput, latency and redundancy. There are a large number of vendors who will conduct data transmission and leakage audits, providing a complete end-to-end data traceability, content verification and inspection.

Step 2: Know Your Partner

Trust but verify.

Your relationship with an outsourced testing partner is by nature an intimate one – exposing your data, business practices, clients and customers, process flows and service offerings. Your relationship with this important partner must be well-defined and predicated on trust, especially when working together in the Cloud.

Tips for defining your relationship with your third-party testing partner:

  • Scope it out – Carefully document your expectations regarding use and limitations of data, data storage, proliferations, etc. Define the compliance expectations, audit approaches, and remediation baselines and thresholds.
  • Lay on the legalese – Obtain a signed non-disclosure confidentiality agreement and other protective legal documentation.
  • Assess capabilities – Validate your testing partner’s capabilities both from a testing capacity as well as a security compliance capacity. Are they located at a secure facility? Do they have internal governance policies that dictate their focus on protecting customer data? Are their computer and network systems capable of providing the encryption and data storage functions needed?
  • Select bonded services – Require that your testing partner is bonded and that they bond their employees. A fidelity ‘Bond’ is usually a guarantee against dishonesty such as data and identity theft. Most fidelity bonds have an arrest and conviction clause and often prove to be a powerful deterrent.

Step 3: Determine Your Model

It is important to map out where your data assets are located, how they interrelate, and how they should be segregated.

  • Role-based systems – Document not only details about the data but details about who uses the data and why. Roles within a process should be allowed only specific uses of data related to defined tasks.
  • Data access model – Firms with lasting outsourcing relationships with testing partners carefully create a data usability map to:
    • Identify users and assign roles / access permissions.
    • Decide architecturally where user authentication will occur. (e.g. user IDs and all attributes known only to the Web server).
    • Define session constraints, Single Instance rules, data volume limits, etc.
    • Decide monitoring functions, alerts, and reports.

Step 4: Audit Activity & Remediate Issues

  • Validate that things are running smoothly on a periodic basis:
    • Collect clickstream information.
    • Determine what level of granularity you want to track and how often (periodicity).
    • Track the types and volumes of testing activity (manual testing vs. automated testing).
    • Report on the activity.
    • Identify the types of users and match names to user accounts and provide reporting of access and usage.
    • Conduct other ongoing protective oversight, monitoring, logging and reporting.
    • Track and report on system availability.
  • Remediate performance and security issues based on expectations (baselines) and boundaries (thresholds) determined in Step 2.

When measuring the success of your partnership with an outsourced testing partner, include these steps as part of a review checklist. Documentation, diligence and transparency from both parties will go a long way to keeping that Cloud locked down tight.

LogiGear Corporation

LogiGear Corporation provides global solutions for software testing, and offers public and corporate software-testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast and cost-effective results. Since 1994, LogiGear has worked with hundreds of companies from the Fortune 500 to early-stage startups, creating unique solutions to exactly meet their needs. With facilities in the US and Vietnam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

For more information, contact Joe Hughes + 01 650.572.1400

LogiGear Corporation
LogiGear Corporation provides global solutions for software testing, and offers public and corporate software testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast, cost-effective results. Since 1994, LogiGear has worked with Fortune 500 companies to early-stage start-ups in, creating unique solutions to meet their clients’ needs. With facilities in the US and Viet Nam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

The Related Post

Music is an important cultural component for any society, and Vietnam is no exception. The history of Vietnamese music is a long and refined one, with influences from both East and West depending on the geopolitical nuances of the time. From Chinese traditional music to American-style pop, musical diversity is one of the country’s most ...
While Vietnamese coffee seems to garner the most attention, there’s another French import that is becoming increasingly popular worldwide – banh mi. Banh mi literally means “bread” though it is often extrapolated to include its sandwich form which is sold on seemingly every corner and in every alleyway in Vietnam’s cities. Bread was introduced by ...
Ingenuity reigns as the Vietnamese prove that cars are second best when it comes to transporting heavy loads and entire families.
Whenever the subject of managing remote teams arises, most leads and managers that I know would prefer that the work remain in their home office.
Rich or poor, rain or shine, hot or cold, chè is one of Vietnam’s most unique culinary offerings. Is it a desert? A breakfast? A snack? Chè, a variety of sweet soup, is one of Vietnam’s most versatile foods, consumed by the old and young, the rich and the poor. There are literally dozens of ...
There’s a reason so many major companies are outsourcing to Vietnam. Intel is building the largest semiconductor plant in the world in Vietnam. IBM, Sony and dozens of other Major corporations have opened centers in Vietnam in the last few years. Business Week calls Vietnam “The new hot spot for IT outsourcing” Gartner has added ...
In 2000, offshoring was not a new concept as many large companies had already been offshoring and outsourcing for over a decade. What did change, in the past 10 years was how many companies began outsourcing and how many development tasks were being distributed. It was not only Texas Instrument and Microsoft sized companies distributing ...
Over the years we’ve provided an extensive number of articles that provide a wealth of knowledge about outsourcing. Below are links to some of those articles. Avoid Epic Fail. Get Professional Help This article covers the benefits of using professional services team to get great test automation quickly.
Introduction Global Test Automation is a strategy that integrates Manual Software Testing, Test Automation, and global resource strategies to maximize the benefits of Software Testing while minimizing the costs. For executives, there are 10 key points to remember when embarking on a strategy of Global Test Automation, which are enumerated in this article.
Test Leads and Test Managers very rarely make the decision to offshore. It is typically not a choice, but rather a mandate from company executives who look to offshoring for significant cost reduction. Among US leads and managers responsible for offshore teams, management and oversight of the offshore teams is now cited as their largest ...
Using an outsourcing firm with technical know-how and sound processes adds value from the very beginning. Flexible outsourcing, in particular offshoring, can make the difference in achieving business goals within the boundaries of time and budget. Organizations rarely have a predictable, steady flow of work. Most test teams are under extreme pressure to get new products ...
Viet Nam is entering its economic ‘Golden Age.’ Will the final picture resemble Japan or Thailand?

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news