Four Fundamental Requirements of Successful Testing in the Cloud – Part II

Internet-based per-use service models are turning things upside down in the software development industry, prompting rapid expansion in the development of some products and measurable reduction in others. (Gartner, August 2008) This global transition toward computing “in the Cloud” introduces a whole new level of challenge when it comes to software testing.

Cloud computing, when it is done well, provides a reliable and single point of access for users. Consistent, positive user experience sells the service, and rigorous testing assures a quality experience. In order to produce reliable, effective results for users of many walks of life, exacting software testing standards must be met.

In a series of articles, LogiGear is identifying four fundamental requirements by which software testing in the Cloud can uphold these standards:

Four Fundamental Requirements for Successful Testing in the Cloud:

01: Constantly Test Everything – the Right Way
02: Know What’s Where – and Prove It
03: Define Your Paradigm
04: Don’t Underestimate the Importance of Positive User Experience

In this issue’s article, we address:

Requirement 02: Know What’s Where – and Prove It

Although some would argue to the contrary, the nature of Cloud computing creates inherent risks not encountered in traditional software development. Financial transactions and the storing of Protected Consumer Information, especially on Internet-connected systems, open the door to potential data losses, undesirable user experiences… and audit compliance issues. Responsibilities once relegated to local IT departments are becoming core competencies for QA organizations. Many companies are finding that it makes sense to bring in an experienced Cloud testing partner to help the in-house team avoid these problems.

Introducing relationships with third-party partners, Cloud-based datacenters and application hosting services adds an additional level of complexity to tracking and demonstrating data security.

Interruptions in Cloud-based services are troubling because companies are increasingly evolving their operations to run critical software as well as storing larger and larger amounts of data in the Cloud computing paradigm. So if service is interrupted, organizations may find that they are unable to function.

For our discussion of the right tools and methodology, read the first installment of this series: Requirement 01: Constantly Test Everything – the Right Way.

There are several approaches major firms take to better insure the utility of the Cloud services they receive.

  1. Insist on service-level agreements – Google offers service level agreements (SLA) promising that components of their offerings will be available as much as 99.9 percent of the time (9 hours per year down time), with service credits if availability drops below the agreed upon levels. The only way companies can realistically achieve this model is to have superior test sets, that are fully automated – and to go about automation the right way. Otherwise it can quickly become unachievable and unmanageable.
  2. Transparency through dashboards – Make use of the Cloud vendor’s dashboards and service interruption alerts. Google, Yahoo, Amazon and other major players give developed administrative dashboards tracking status availabilities, downtimes, notifications and alerts. shows the response times for their server transactions. They detail problems, their affects on system availability and other related applications, and, more importantly what caused the problems and what the mitigations are.
  3. Cloud monitoring services – This emerging sector of the Cloud services spectrum dives deeper into the details of Cloud availability, status, service interruptions and problem mitigations.
  4. Automate Cloud testing – As you integrate a variety of applications on the Cloud it’s important to be able to quickly test your work flows when introducing a new or revised app.

In addition to these approaches, asking four key questions can assist in mitigating some of the potential outsourcing risks and help drive a more secure and predictable partnership:

  1. Know Yourself
  2. Know Your Partner
  3. Determine Your Model
  4. Audit Activity & Remediate Issues

Step 1: Know Yourself

Detailed understanding of your application, data usage, security and risk profile is necessary when preparing to engage a third-party testing partner. Here are some tips to take into consideration as you assemble a list of requirements:

Application Architecture:

  • Performance – Testing a complex Cloud application can be complicated by the response latency of sites containing huge objects. Be sure to anticipate what any performance bottleneck might be with your outsourced testing partner, as even small delays can have significant impact on the efficiency of their testing.
  • Direct connect – It is important to be aware of all of the components that make up the connection between you, your outsourced testing partner and your Cloud-based service provider – not just for efficiency’s sake, but to make sure that your data doesn’t take any unwanted detours between here and there.
A broad-reaching 12-month survey by Orthos Corp found that “…every organization without exception had suffered multiple instances of data leakage, many of them serious and potentially damaging.”

Data Architecture:

  • Data in transit – Encrypting data in transit is far more challenging. Be sure to identify or implement specific products and strategies to encrypt data so that it cannot be read or tampered with as it crosses the networks between you and your outsourced testing partner.

Systems Architecture:

  • Map out the route – Document each stopping and transition point on the route data takes across networks during testing activity. Ensure that data are protected from the beginning of the journey to the journey’s end. Constructing an effective strategy to protect your data involves knowing where that data travels to and what happens to it along the way.
  • Evaluate efficiency – Ensure that your Cloud partner is not paying a prohibitive performance penalty due to accessing, unencrypting or otherwise unwinding the security protection wrapping your data. Take into consideration available and accessible network bandwidths (effective end-to-end bandwidth), and any data or application and query transaction latencies.
  • Audit for leakage – Look beyond available bandwidth to see what the effective capabilities are with regard to throughput, latency and redundancy. There are a large number of vendors who will conduct data transmission and leakage audits, providing a complete end-to-end data traceability, content verification and inspection.

Step 2: Know Your Partner

Trust but verify.

Your relationship with an outsourced testing partner is by nature an intimate one – exposing your data, business practices, clients and customers, process flows and service offerings. Your relationship with this important partner must be well-defined and predicated on trust, especially when working together in the Cloud.

Tips for defining your relationship with your third-party testing partner:

  • Scope it out – Carefully document your expectations regarding use and limitations of data, data storage, proliferations, etc. Define the compliance expectations, audit approaches, and remediation baselines and thresholds.
  • Lay on the legalese – Obtain a signed non-disclosure confidentiality agreement and other protective legal documentation.
  • Assess capabilities – Validate your testing partner’s capabilities both from a testing capacity as well as a security compliance capacity. Are they located at a secure facility? Do they have internal governance policies that dictate their focus on protecting customer data? Are their computer and network systems capable of providing the encryption and data storage functions needed?
  • Select bonded services – Require that your testing partner is bonded and that they bond their employees. A fidelity ‘Bond’ is usually a guarantee against dishonesty such as data and identity theft. Most fidelity bonds have an arrest and conviction clause and often prove to be a powerful deterrent.

Step 3: Determine Your Model

It is important to map out where your data assets are located, how they interrelate, and how they should be segregated.

  • Role-based systems – Document not only details about the data but details about who uses the data and why. Roles within a process should be allowed only specific uses of data related to defined tasks.
  • Data access model – Firms with lasting outsourcing relationships with testing partners carefully create a data usability map to:
    • Identify users and assign roles / access permissions.
    • Decide architecturally where user authentication will occur. (e.g. user IDs and all attributes known only to the Web server).
    • Define session constraints, Single Instance rules, data volume limits, etc.
    • Decide monitoring functions, alerts, and reports.

Step 4: Audit Activity & Remediate Issues

  • Validate that things are running smoothly on a periodic basis:
    • Collect clickstream information.
    • Determine what level of granularity you want to track and how often (periodicity).
    • Track the types and volumes of testing activity (manual testing vs. automated testing).
    • Report on the activity.
    • Identify the types of users and match names to user accounts and provide reporting of access and usage.
    • Conduct other ongoing protective oversight, monitoring, logging and reporting.
    • Track and report on system availability.
  • Remediate performance and security issues based on expectations (baselines) and boundaries (thresholds) determined in Step 2.

When measuring the success of your partnership with an outsourced testing partner, include these steps as part of a review checklist. Documentation, diligence and transparency from both parties will go a long way to keeping that Cloud locked down tight.

LogiGear Corporation

LogiGear Corporation provides global solutions for software testing, and offers public and corporate software-testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast and cost-effective results. Since 1994, LogiGear has worked with hundreds of companies from the Fortune 500 to early-stage startups, creating unique solutions to exactly meet their needs. With facilities in the US and Vietnam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

For more information, contact Joe Hughes + 01 650.572.1400

LogiGear Corporation
LogiGear Corporation provides global solutions for software testing, and offers public and corporate software testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast, cost-effective results. Since 1994, LogiGear has worked with Fortune 500 companies to early-stage start-ups in, creating unique solutions to meet their clients’ needs. With facilities in the US and Viet Nam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

The Related Post

Authors: Hung Q. Nguyen, Michael Hackett, Brent K. Whitlock Paperback: 164 pages Publisher: Happy About (August 1, 2006) Language: English Product Dimensions: 8.4 x 5.1 x 0.5 inches “Software is complex but I’m tired of finding bug after bug that a 5th grader would have turned in. Virtually every technical product these days includes a ...
While Vietnamese coffee seems to garner the most attention, there’s another French import that is becoming increasingly popular worldwide – banh mi. Banh mi literally means “bread” though it is often extrapolated to include its sandwich form which is sold on seemingly every corner and in every alleyway in Vietnam’s cities. Bread was introduced by ...
LogiGear Magazine December Outsourcing Reinvented Issue 2017
Over the years we’ve provided an extensive number of articles that provide a wealth of knowledge about outsourcing. Below are links to some of those articles. Avoid Epic Fail. Get Professional Help This article covers the benefits of using professional services team to get great test automation quickly.
BJ Rollison, Software Test Architect for Microsoft, is planning to speak at the Viet Nam International Software Testing Automation Conference 2010 (VISTACON 2010) hosted by LogiGear Corporation from 20 – 22 September 2010. Mr. Rollison started working for Microsoft in 1994, becoming one of the leading experts of test architecture and execution at Microsoft. He ...
Hong Mai has always been interested in computers since the beginning. She recalls having written her first script and thought it was amazing. “Writing a script and having it run successfully is a good feeling,” says Hong Mai. Hong Mai’s parents were high school teachers but have since retired. It was certain that their two ...
‘The Pearl of the Orient’ boasts some of SE Asia’s best examples of historic architecture. While much has been lost, the city still offers a plethora of beautiful old buildings. Saigon, once known as ‘Pearl of the Orient’ claims some of South East Asia’s most stunning historic buildings. While lack of preservation laws have resulted ...
Tết holidays is a celebration of family, food and a favorable new year. February third celebrates Tết Nguyên Đán, or otherwise known as Tết, the Lunar New Year holiday welcoming the year of the cat. The holiday varies from late January to early February officially lasting three days. More recently, China and Vietnam celebrate Tết ...
Have you ever tried to push heterogeneous groups of people to move in the same direction? It is quite a challenge, isn’t it? And now, imagine this situation in a distributed environment, with people whom you’ve never met, and who must be persuaded by means of phone calls or email. Welcome to our world. We have ...
LogiGear Magazine – September 2011 – The Offshoring and Outsourcing Issue
Jamie Tischart Director – Sews, Product Delivery McAfee, Inc. 9781 S. Meridian Blvd, Suite 400 Englewood, Co, USA, 80112 LogiGear: What do you think about the development of the software testing industry in Vietnam? Jamie Tischart: I am very excited about the continued growth and development of the software testing industry in Vietnam. The professionals ...
Ethics in IT Outsourcing by Tandy Gold is a surprisingly great read on the ethics in IT outsourcing as well as IT ethics and business ethics more generally. The reason it was a surprise is that the Gold tackles the ethics in IT outsourcing head on. From its roots in US economics and public policy ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news