Four Fundamental Requirements of Successful Testing in the Cloud – Part II

Internet-based per-use service models are turning things upside down in the software development industry, prompting rapid expansion in the development of some products and measurable reduction in others. (Gartner, August 2008) This global transition toward computing “in the Cloud” introduces a whole new level of challenge when it comes to software testing.

Cloud computing, when it is done well, provides a reliable and single point of access for users. Consistent, positive user experience sells the service, and rigorous testing assures a quality experience. In order to produce reliable, effective results for users of many walks of life, exacting software testing standards must be met.

In a series of articles, LogiGear is identifying four fundamental requirements by which software testing in the Cloud can uphold these standards:

Four Fundamental Requirements for Successful Testing in the Cloud:

01: Constantly Test Everything – the Right Way
02: Know What’s Where – and Prove It
03: Define Your Paradigm
04: Don’t Underestimate the Importance of Positive User Experience

In this issue’s article, we address:

Requirement 02: Know What’s Where – and Prove It

Although some would argue to the contrary, the nature of Cloud computing creates inherent risks not encountered in traditional software development. Financial transactions and the storing of Protected Consumer Information, especially on Internet-connected systems, open the door to potential data losses, undesirable user experiences… and audit compliance issues. Responsibilities once relegated to local IT departments are becoming core competencies for QA organizations. Many companies are finding that it makes sense to bring in an experienced Cloud testing partner to help the in-house team avoid these problems.

Introducing relationships with third-party partners, Cloud-based datacenters and application hosting services adds an additional level of complexity to tracking and demonstrating data security.

Interruptions in Cloud-based services are troubling because companies are increasingly evolving their operations to run critical software as well as storing larger and larger amounts of data in the Cloud computing paradigm. So if service is interrupted, organizations may find that they are unable to function.

For our discussion of the right tools and methodology, read the first installment of this series: Requirement 01: Constantly Test Everything – the Right Way.

There are several approaches major firms take to better insure the utility of the Cloud services they receive.

  1. Insist on service-level agreements – Google offers service level agreements (SLA) promising that components of their offerings will be available as much as 99.9 percent of the time (9 hours per year down time), with service credits if availability drops below the agreed upon levels. The only way companies can realistically achieve this model is to have superior test sets, that are fully automated – and to go about automation the right way. Otherwise it can quickly become unachievable and unmanageable.
  2. Transparency through dashboards – Make use of the Cloud vendor’s dashboards and service interruption alerts. Google, Yahoo, Amazon and other major players give developed administrative dashboards tracking status availabilities, downtimes, notifications and alerts. shows the response times for their server transactions. They detail problems, their affects on system availability and other related applications, and, more importantly what caused the problems and what the mitigations are.
  3. Cloud monitoring services – This emerging sector of the Cloud services spectrum dives deeper into the details of Cloud availability, status, service interruptions and problem mitigations.
  4. Automate Cloud testing – As you integrate a variety of applications on the Cloud it’s important to be able to quickly test your work flows when introducing a new or revised app.

In addition to these approaches, asking four key questions can assist in mitigating some of the potential outsourcing risks and help drive a more secure and predictable partnership:

  1. Know Yourself
  2. Know Your Partner
  3. Determine Your Model
  4. Audit Activity & Remediate Issues

Step 1: Know Yourself

Detailed understanding of your application, data usage, security and risk profile is necessary when preparing to engage a third-party testing partner. Here are some tips to take into consideration as you assemble a list of requirements:

Application Architecture:

  • Performance – Testing a complex Cloud application can be complicated by the response latency of sites containing huge objects. Be sure to anticipate what any performance bottleneck might be with your outsourced testing partner, as even small delays can have significant impact on the efficiency of their testing.
  • Direct connect – It is important to be aware of all of the components that make up the connection between you, your outsourced testing partner and your Cloud-based service provider – not just for efficiency’s sake, but to make sure that your data doesn’t take any unwanted detours between here and there.
A broad-reaching 12-month survey by Orthos Corp found that “…every organization without exception had suffered multiple instances of data leakage, many of them serious and potentially damaging.”

Data Architecture:

  • Data in transit – Encrypting data in transit is far more challenging. Be sure to identify or implement specific products and strategies to encrypt data so that it cannot be read or tampered with as it crosses the networks between you and your outsourced testing partner.

Systems Architecture:

  • Map out the route – Document each stopping and transition point on the route data takes across networks during testing activity. Ensure that data are protected from the beginning of the journey to the journey’s end. Constructing an effective strategy to protect your data involves knowing where that data travels to and what happens to it along the way.
  • Evaluate efficiency – Ensure that your Cloud partner is not paying a prohibitive performance penalty due to accessing, unencrypting or otherwise unwinding the security protection wrapping your data. Take into consideration available and accessible network bandwidths (effective end-to-end bandwidth), and any data or application and query transaction latencies.
  • Audit for leakage – Look beyond available bandwidth to see what the effective capabilities are with regard to throughput, latency and redundancy. There are a large number of vendors who will conduct data transmission and leakage audits, providing a complete end-to-end data traceability, content verification and inspection.

Step 2: Know Your Partner

Trust but verify.

Your relationship with an outsourced testing partner is by nature an intimate one – exposing your data, business practices, clients and customers, process flows and service offerings. Your relationship with this important partner must be well-defined and predicated on trust, especially when working together in the Cloud.

Tips for defining your relationship with your third-party testing partner:

  • Scope it out – Carefully document your expectations regarding use and limitations of data, data storage, proliferations, etc. Define the compliance expectations, audit approaches, and remediation baselines and thresholds.
  • Lay on the legalese – Obtain a signed non-disclosure confidentiality agreement and other protective legal documentation.
  • Assess capabilities – Validate your testing partner’s capabilities both from a testing capacity as well as a security compliance capacity. Are they located at a secure facility? Do they have internal governance policies that dictate their focus on protecting customer data? Are their computer and network systems capable of providing the encryption and data storage functions needed?
  • Select bonded services – Require that your testing partner is bonded and that they bond their employees. A fidelity ‘Bond’ is usually a guarantee against dishonesty such as data and identity theft. Most fidelity bonds have an arrest and conviction clause and often prove to be a powerful deterrent.

Step 3: Determine Your Model

It is important to map out where your data assets are located, how they interrelate, and how they should be segregated.

  • Role-based systems – Document not only details about the data but details about who uses the data and why. Roles within a process should be allowed only specific uses of data related to defined tasks.
  • Data access model – Firms with lasting outsourcing relationships with testing partners carefully create a data usability map to:
    • Identify users and assign roles / access permissions.
    • Decide architecturally where user authentication will occur. (e.g. user IDs and all attributes known only to the Web server).
    • Define session constraints, Single Instance rules, data volume limits, etc.
    • Decide monitoring functions, alerts, and reports.

Step 4: Audit Activity & Remediate Issues

  • Validate that things are running smoothly on a periodic basis:
    • Collect clickstream information.
    • Determine what level of granularity you want to track and how often (periodicity).
    • Track the types and volumes of testing activity (manual testing vs. automated testing).
    • Report on the activity.
    • Identify the types of users and match names to user accounts and provide reporting of access and usage.
    • Conduct other ongoing protective oversight, monitoring, logging and reporting.
    • Track and report on system availability.
  • Remediate performance and security issues based on expectations (baselines) and boundaries (thresholds) determined in Step 2.

When measuring the success of your partnership with an outsourced testing partner, include these steps as part of a review checklist. Documentation, diligence and transparency from both parties will go a long way to keeping that Cloud locked down tight.

LogiGear Corporation

LogiGear Corporation provides global solutions for software testing, and offers public and corporate software-testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast and cost-effective results. Since 1994, LogiGear has worked with hundreds of companies from the Fortune 500 to early-stage startups, creating unique solutions to exactly meet their needs. With facilities in the US and Vietnam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

For more information, contact Joe Hughes + 01 650.572.1400

LogiGear Corporation
LogiGear Corporation provides global solutions for software testing, and offers public and corporate software testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast, cost-effective results. Since 1994, LogiGear has worked with Fortune 500 companies to early-stage start-ups in, creating unique solutions to meet their clients’ needs. With facilities in the US and Viet Nam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

The Related Post

Vietnam draws thousands of foodies each year who are eager to explore the country’s large selection of fresh and simple food. The options are nearly endless, and it’s no exaggeration to say one would need to invest a few years in order to sample everything. Vietnamese cuisine in general has clear regional distinctions. Hanoi is ...
LogiGear Magazine December Outsourcing Reinvented Issue 2017
Tết holidays is a celebration of family, food and a favorable new year. February third celebrates Tết Nguyên Đán, or otherwise known as Tết, the Lunar New Year holiday welcoming the year of the cat. The holiday varies from late January to early February officially lasting three days. More recently, China and Vietnam celebrate Tết ...
Whether you’re looking for a 5-star resort or a UNESCO World Heritage site,Vietnam’s 2025 mile coastline has something for everyone. Vietnam is a country with a plethora of natural beauty which manifests in many different ways. And with 2025 miles of coastline, the country’s beaches are perhaps the pinnacle of that beauty. The characteristics of these ...
  Amidst the populated sidewalks with countless motorbikes sprawling over the streets like water trying to run through rocks, Vietnamese cities are densely packed with chaotic movement. Yet, in all its overwhelming energy, the Vietnamese have definitely taken on the tradition of its once French colonists: drinking coffee.
Whenever the subject of managing remote teams arises, most leads and managers that I know would prefer that the work remain in their home office.
Well publicized offshore outsourcing challenges, a narrowing labor-cost gap, and political considerations have some rethinking how to approach outsourcing. Enterprises elect to bring offshore or outsourced operations in-house for a number of reasons. While performance certainly can play a role, motivation also includes strategic business reasons and a belief that the enterprise can perform the function ...
Over the years we’ve provided an extensive number of articles that provide a wealth of knowledge about outsourcing. Below are links to some of those articles. Avoid Epic Fail. Get Professional Help This article covers the benefits of using professional services team to get great test automation quickly.
McAfee representatives from India and Japan visited LogiGear Viet Nam from Feb 21 – 25. Their week-long stay entailed debriefing LogiGear staff on McAfee MX Logic and Mobile Security for the team to begin testing. Project manager Tien Nguyen and project lead Hung Le oversaw the sessions between LogiGear and McAfee. As a good hostess, ...
Test Leads and Test Managers very rarely make the decision to offshore. It is typically not a choice, but rather a mandate from company executives who look to offshoring for significant cost reduction. Among US leads and managers responsible for offshore teams, management and oversight of the offshore teams is now cited as their largest ...
BJ Rollison, Software Test Architect for Microsoft, is planning to speak at the Viet Nam International Software Testing Automation Conference 2010 (VISTACON 2010) hosted by LogiGear Corporation from 20 – 22 September 2010. Mr. Rollison started working for Microsoft in 1994, becoming one of the leading experts of test architecture and execution at Microsoft. He ...
While Vietnamese coffee seems to garner the most attention, there’s another French import that is becoming increasingly popular worldwide – banh mi. Banh mi literally means “bread” though it is often extrapolated to include its sandwich form which is sold on seemingly every corner and in every alleyway in Vietnam’s cities. Bread was introduced by ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news