Four Fundamental Requirements of Successful Testing in the Cloud – Part III

Internet-based per-use service models are turning things upside down in the software development industry, prompting rapid expansion in the development of some products and measurable reduction in others. (Gartner, August 2008) This global transition toward computing “in the Cloud” introduces a whole new level of challenge when it comes to software testing.

Cloud computing, when it is done well, provides a reliable and single point of access for users. Consistent, positive user experience sells the service, and rigorous testing assures a quality experience. In order to produce reliable, effective results for users of many walks of life, exacting software testing standards must be met.

In a series of articles, LogiGear is identifying four fundamental requirements by which software testing in the Cloud can uphold these standards:

Four Fundamental Requirements for Successful Testing in the Cloud:

  1. Constantly Test Everything – the Right Way
  2. Know What’s Where – and Prove It
  3. Define Your Paradigm
  4. Don’t Underestimate the Importance of Positive User Experience

In this issue’s article, we address:

Requirement 03: Define your Paradigm

Today’s companies increasingly find that they are in an ever more competitive market, especially in the drive to implement more robust, capable and pioneering Cloud-based products and services. Product delivery times are decreasing, customers demand higher and higher levels of product quality, and failure to deliver within the customer’s expectations can be swiftly punished with whole scale product abandonment and erection of barriers to market reentry.

Companies who are leading creative efforts to address these volatile areas recognize that quality must be uncompromised, and indeed surpassed in every way.

Adopting a testing paradigm that is designed specifically for the requirements of Cloud-computing is a fundamental requirement for the new standards of quality being set by customer-driven demand.

The steps for defining a Cloud-friendly testing paradigm are in general:

  1. Evaluate Your Current Testing Paradigm
  2. Define Paradigm Requirements that Meet Standards for Cloud-based Technologies
  3. Apply Proven Methods and Technology
For our discussion of the right tools and methodology, read the first installment of this series: Requirement 01: Constantly Test Everything – the Right Way.

Our discussion about loss prevention and risk mitigation when testing in the Cloud can be read in the second series installment: Requirement 02: Know What’s Were – and Prove It

1. Evaluate Your Current Testing Paradigm

Each quality or development team has its own frame of reference by which it identifies the need for testing overall, and the specific kinds of testing required. This frame of reference, or paradigm, often reflects the higher organizational approach to quality assurance and the role that testing is perceived to occupy in customer satisfaction and loyalty.

Cloud computing introduces a need for a new paradigm – one that incorporates the 24/7 testing-on-the-fly required for successful Software as a Service implementation and other Cloud-based products.

Some organizations may find their current approach to testing adapts relatively well for the “test everything all the time” model. However, most will find it necessary to reassess not just their approach toward testing, but the fundamental principles that underlie that approach – ones they’ve inherited from traditional testing models.

For a more detailed look at the historical development of testing methodologies, read our white paper “The 5% Solution”

In the world of software testing, all the players used to start in pretty much the same place: manual testing. As the industry matured, more elegant testing solutions – record and playback, automated or scripted testing, and action-based testing – emerged over time. Each new approach to testing successively increased our testing efficiencies, reporting efficiencies and ability to handle larger and larger test loads, lending leverage to the leading few. Conducting a Current-State Assessment can give you the necessary insight into what works and doesn’t work in your current testing paradigm when it comes to testing in the Cloud.

Conducting a Current-State Assessment

Most companies have tried a variety of testing approaches, some manual, some scripted, and maybe even some attempts at automation. It is important for a firm to conduct an honest self-assessment about their current testing footprint, capabilities, limitations, and opportunities for improvement.

Many times firm have a belief about their current testing approach and are surprised to find that those beliefs, albeit hopeful, are not fully rooted in practice. The following questions will help thoroughly evaluate your testing paradigm.

  • Testing methodology – Is it largely manual or some combination of methods? Are those methods consistent?
  • Testing environment – Where are the applications located? Where is the data located? What are the network and performance constructs?
  • People resources and skill sets – What is the competency makeup of your current test team? What are the skill sets required for moving forward?
  • Time-to-deliver constraints – What are the barriers to testing efficiency? What are the efforts that take more time than others and why?
  • Reporting and progress visibility – Do you have consistent visibility into your testing status? Are you surprised when testing efforts are late? Are statuses verbal, or based on actual metrics?

2. Define Paradigm Requirements that Meet Standards for Cloud-based Technologies

Achieving your organization’s testing goals, in whole or in part, requires planning and mapping out your “testing in the Cloud” strategy and its requirements.

Clarifying Cloud Architecture

An initial understanding of the different types of clouds and what their high-level architectures offer will inform the kind of testing paradigm you will need to establish.

Sam Johnston’s integrated picture depicts the three cloud types:

  • Remote, Public or External Cloud – Public clouds are sometimes referred to as “Regular” cloud computing. Completely separate from a user’s desktop or the corporate network that they belong to, public clouds offer a pay-per-use service model because the user is leveraging outside compute resources for the particular service they are seeking.This approach offers economies of scale, but their shared infrastructure model can raise concerns about configuration restrictions, adequate security, and service-levels (available up-time). These concerns might make you think twice about subjecting sensitive data that is subject to compliance or safe harbor regulations.Because “public” clouds are typically made available via the public internet they may be free or inexpensive to use. A well-known example of a public cloud is Amazon EC2 which is available for use by the general public.
  • Internal or Private Cloud – Private cloud computing extends the same infrastructure concepts firms already have in their data centers. The motivation for private clouds appears to be to resolve security and availability concerns inherent in the public cloud paradigm.As such, private clouds seemingly are not burdened by network bandwidth, availability issues or potential security risks that may be associated with public clouds. However, this thinking belies the very intent of cloud computing which is predicated on hardware-software extensibility, dramatic reduction in infrastructure costs, and an elimination of the management concerns governing private networks.
  • Mixed or Hybrid Cloud – Many of the leading engineering thinkers in the industry suggest that the most workable cloud computing approach is the “hybrid” approach. The hybrid solution combines the best of the Public and Private Cloud paradigms.Considering that some applications may be too complex or too sensitive to risk operating from a public cloud it makes sense for a firm to protect those application and data assets within the construct of a private cloud where they have total control.Less sensitive applications and data can be migrated to a public cloud freeing compute resources that can be re-purposed for the complex applications that need to stay home.The hybrid approach does sound like the best of both worlds. It makes sense from a technology and economic standpoint. It allows for control, flexibility and growth. The trick to managing hybrid clouds comes when you consider spikes in demand.When demand spikes pummel the performance of your applications located within the private cloud -and you need additional computing power (such as is experienced by web-based news media when critical events occur) -you will need to develop a management policy that can be responsible for when to reach out to the public cloud for those additional resources.

Defining the Future-State

Envisioning the results of your testing transformation requires solid understanding of your organization’s business goals and objectives, the Cloud computing paradigms that may help your testing effort contribute to those goals, and development of a sound plan to move in the new direction.

When documenting your planned future state, address each of these categories:

  • Architectural – Consider the different Cloud paradigms as they pertain to your business model, goals and objectives, and application and data sensitivity.
  • Organization – Ensure organizational review and consensus of new Cloud testing direction as it pertains to business goals and objectives and priorities.
  • Financial – Define the benefits, know where the real costs lie, and define the budget.
  • Implementation – Adopt an incremental improvement approach, and choose the correct tools and partners.
  • Monitor and measure – Develop a consistent set of metrics for measuring and monitoring your new foray into testing in the Cloud.

Other organizational goals and objectives that are important to include:

  • Provide sufficient support for distributed test teams
  • Increase your return-on-testing-investment
  • Significantly decrease time-to-market
  • Optimize the reusability of tests and test automation
  • Improve test output and coverage
  • Enhance the motivation of your testing staff
  • Increase managerial control over quality and testing
  • Have better visibility into quality and testing effectiveness

While setting your testing goals, consider that in the LogiGear white paper “The 5% Solution,” Hans Buwalda, LogiGear’s CTO, posits that “with good test automation you can have more tests and execute them any time you need to, thus significantly improving the development cycles of your project.”

With the repeatable tests under automation, he says, testers are free to work on more sophisticated testing as well as testing focused on new features and functions increasing the overall quality of the delivered result.

Buwalda establishes the 5% Goals:

  • No more than 5% of structured test cases should be tested manually
  • No more than 5% of testing efforts should be spent in the automation process

Conversely, the five percent goals suggest that 95% of your tests should be automated and 95% of your testing efforts should be spent on emerging functionality and higher complexity testing. Other organizational goals fall into some pretty predictable categories.

3. Apply Proven Methods and Technology

Implementing the changes necessary to adopt a new testing paradigm will bear out how well you’ve defined your requirements. Therefore, selecting well-matched testing tools and specific architectural approaches can have a dramatic impact on the results of your paradigm shift.

Architectural Approaches

If you are going to perform testing in the cloud it is important to understand the different architectural approaches that are available and the unique advantages that each approach offers. First, let’s start out with a quick list of the six basic cloud architectures or relationships.

Cloud Services Paradigm Industry Standards
Cloud-accessible application
  • Communications (HTTP, XMPP)
  • Security (OAuth, OpenID, SSL/TLS[83])
  • Syndication (Atom)
Cloud-based client services
  • Browsers (AJAX)
  • Offline (HTML 5)
Scalable infrastructure and computational arrays
  • Virtualization (OVF[84])
Application development and deployment platforms
  • Solution stacks (LAMP)
Utility services
  • Data (XML, JSON)
  • Web Services (REST)
Data storage, duplication and backup

Cloud Accessible Application

This category of cloud computing technology focuses on applications that are entirely externally hosted at a cloud services application vendor. No installation of software is required on the user desktop. The application functionality is utilized through a web browser and the data is stored on the application host’s servers. is a good example of this type of application service.

Cloud-based Client Services

Think of a device that is completely useless without a connection to internet services (like the iPhone) and you will have an understanding of what cloud-based client services are all about. Netbooks are another emerging computing approach predicated on utilizing more cloud-based applications and services using a smaller, less powerful laptop computer.

Scalable Infrastructure and Computational Arrays

Imagine a sudden spike in demand for products you want to sell. Good news, right? But let’s image that your computer room server just can’t handle the additional load. You lose money with every customer who abandons their purchase. With the right cloud infrastructure provider relationship those worries might be a thing of the past. Vendors are increasing building expanding infrastructure architectural offers to support this very problem. Good examples to keep in mind are Amazon and Sun Microsystems.

Application Development and Deployment Platforms

Picture the scenario that you have developed a ‘killer’ application that you know everyone is going to want but you can’t deal with the cost and complexity of buying and managing the server and network equipment to manage it. Web hosting services are a good example of this approach allowing individuals and organizations to provide their own web site accessible via the web. Still other platforms allow for remote applications development and deployment.

Utility Services

MapQuest is popular example of a cloud-based service utility. It is completely inaccessible and unusable if you are not connected to the web. Other examples include, PayPal and Yahoo search.

Data Storage, Duplication and Backup

There are a number of variations of data services available via the cloud. Simple data storage allows for a pay-as-you-increase sort of service allowing ever increasing disk utilization as needed. Other services offer data duplication and mirroring, and data backup.


In this paper, we’ve presented the need to “define your paradigm” as a necessary prerequisite for pursuing a strategy for “testing-in-the-cloud”. We began with the common-sense directive of evaluating your current testing approach and setup. We then reviewed the different cloud types, their focus, and their typical use. Additionally, we offered the business objectives most firms would consider as foundation for making a move to a Cloud-based testing paradigm. Lastly, we presented the six common cloud computing paradigms, the industry standards that have emerged to support each, and introduced examples of services in each category.

Your testing-in-the-Cloud strategy should consider all of these approaches to determine what will work best for your organization. Your success is will be determined by a disciplined approach and thorough implementation. Happy testing!

LogiGear Corporation

LogiGear Corporation provides global solutions for software testing, and offers public and corporate software-testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast and cost-effective results. Since 1994, LogiGear has worked with hundreds of companies from the Fortune 500 to early-stage startups, creating unique solutions to exactly meet their needs. With facilities in the US and Vietnam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

For more information, contact Joe Hughes + 01 650.572.1400

LogiGear Corporation
LogiGear Corporation provides global solutions for software testing, and offers public and corporate software testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast, cost-effective results. Since 1994, LogiGear has worked with Fortune 500 companies to early-stage start-ups in, creating unique solutions to meet their clients’ needs. With facilities in the US and Viet Nam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

The Related Post

This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives. Introduction Metrics are the means by which the software quality can be measured; they give you confidence in the product. You may consider these product management indicators, which can be either quantitative or qualitative. They are ...
This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives. Introduction When thinking of the types of Software Testing, many mistakenly equate the mechanism by which the testing is performed with types of Software Testing. The mechanism simply refers to whether you are using Manual or ...
Are you looking for the best books on software testing methods? Here are 4 books that should be on your reading list! The Way of the Web Tester: A Beginner’s Guide to Automating Tests By Jonathan Rasmusson Whether you’re a traditional software tester, a developer, or a team lead, this is the book for you! It ...
In software testing, we need to devise an approach that features a gradual progression from the simplest criteria of testing to more sophisticated criteria. We do this via many planned and structured steps, each of which brings incremental benefits to the project as a whole. By this means, as a tester masters each skill or area ...
Test design is the single biggest contributor to success in software testing. Not only can good test design result in good coverage, it is also a major contributor to efficiency. The principle of test design should be “lean and mean.” The tests should be of a manageable size and at the same time complete and ...
With this edition of LogiGear Magazine, we introduce a new feature, Mind Map. A mind map is a diagram, usually devoted to a single concept, used to visually organize related information, often in a hierarchical or interconnected, web-like fashion. This edition’s mind map, created by Sudhamshu Rao, focuses on tools that are available to help ...
Please note: This article was adapted from a blog posting in Karen N. Johnson’s blog on July 24, 2007. Introduction The password field is one data entry field that needs special attention when testing an application. The password field can be important (since accessing someone’s account can start a security leak), testers should spend more ...
It’s a bird! It’s a plane! It’s a software defect of epic proportions.
Test organizations continue to undergo rapid transformation as demands grow for testing efficiencies. Functional test automation is often seen as a way to increase the overall efficiency of functional and system tests. How can a test organization stage itself for functional test automation before an investment in test automation has even been made? Further, how ...
When it is out of the question to delay delivery, the solution is a prioritization strategy in order to do the best possible job within the time constraints. The scenario is as follows: You are the test manager. You made a plan and a budget for testing. Your plans were, as far as you know, ...
With complex software systems, you can never test all of the functionality in all of the conditions that your customers will see. Start with this as a fact: You will never test enough! Step 2 in getting started is to read and re-read The Art of Software Testing by Glenford Myers. This classic will set the ...
Trying to understand why fails, errors, or warnings occur in your automated tests can be quite frustrating. TestArchitect relieves this pain.  Debugging blindly can be tedious work—especially when your test tool does most of its work through the user interface (UI). Moreover, bugs can sometimes be hard to replicate when single-stepping through a test procedure. ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news