An Interview with Skytap’s Sumit Mehrotra

…On what you need to know before making the transition to EaaS

LogiGear_Skytap's Sumit

1. What are the main differences between cloud-based environments and cloud infrastructure?

An environment is a collection of infrastructure elements working in conjunction to enable an application stack to work.

For example, a simple 3-tier application, with a web front-end component, a business logic component and a database component requires, at a minimum, the following infrastructure components: a few VMs, a few storage disks, and a network. All of these components are required, and they need to work together as an environment for the application to be functional. Note, that as the complexity of the application stack grows, the definition of an environment grows bigger than just infrastructure components. It can include VPNs, public IP addresses, an object store, and other service end-points.

From an application development and testing perspective, the application environment is the starting point of productive work. However, we see teams spending a large amount of time just getting the infrastructure components to a working state and hence losing out on a good portion of their productive work. Working with environments as a unit of work can enable these teams to be more productive and hence service customer and business requirements at a faster pace with higher quality.

2. What is “environment or infrastructure as code?” How can we use this idea in testing?

As the name suggests, infrastructure-as-Code is practice of representing infrastructure as code blocks so that they can be executed whenever needed to build an application stack (environment) in a repeatable and consistent manner. Depending on the actual tool used, it can allow you a variety of things including:

  • Deploy basic infrastructure elements, VMs, disks, networks, etc.
  • Install and configure operating systems
  • Deploy and configure application components
  • Populate application data

With a repeatable and consistent process, bugs can be reproduced, fixed, and validated efficiently. Developers can recreate the same environment that a tester is using to find bugs by using the same Infrastructure-as-Code scripts as the tester is using. This practice goes a long way in solving the problem of ‘it-works-on-my-machine’ syndrome.

There are various paradigms available to create consistent environments in a repeatable manner. Two such popular paradigms are “patterns” and “clones.” With patterns, you can create the environment from scratch each time—allowing you to progress the same exact code at each point of your software delivery pipeline. But it takes time for the environment to be provisioned and can be error-prone especially, for very complex environments. With clones, you can create another copy of the environment from a pre-existing copy. The process can be very fast, even for complex environments. However, moving the same clone through your SDLC can be a heavyweight process.

There is no right or wrong paradigm here. Successful teams have chosen a combination of paradigms and have made trade-offs between time and tooling consistency, to achieve the velocity they need in their software delivery lifecycle.

3. How do companies use cloud-based environments to facilitate testing in DevOps?

Testing especially in the DevOps world has the following key requirements:

  • Instant self-service access to application stacks for testing
  • Support the complexity and scale of the application stack needed at various stages of testing
  • Consistent and reproducible ways to create test environments
  • Ability to collaborate and share environments with developers and testers as part of a feedback loop. Constant feedback is a key aspect of Agile development
  • Ability to ensure that the right resources are available at any time and are being utilized judiciously

All of these requirements are hard to meet without a cloud-based platform. So, cloud-based environments are essential to facilitate testing workflows.

4We know that getting the right environments for testing can be difficult, and sometimes it’s out of a Test team’s control. How are innovative organizations solving this challenge?


Organizations are increasingly turning to the cloud for on-demand and self-service access test environments, but not every cloud platform is suited for the needs of the applications in an organization’s portfolio. Innovative organizations evaluate the environment needs of their applications, and some of the criteria that we have seen customers use are as follows:

  • Configurability – Can the platform deliver test environments that capture the complexity of my application at each stage of testing?
  • Consistency – Can the platform deliver test environments that are in the exact state that we need them to be, whenever I want?
  • Collaboration – Can the platform make it easier for my team (Devs + QA + Ops) to work together more productively?
  • Control – Does the platform have controls to ensure the right people have the appropriate resources to do their jobs?

5. How long does it take to build these environments? Some Test teams do not have dedicated IT members—does the Test team often do this?

The time to get a fully configured environment to test can vary in range from a few minutes to a few weeks. The availability of cloud platforms, and especially IaaS has reduced the far-end of the range to a few days primarily by taking the acquisition of hardware out of the equation. This is the lowest layer of the environment and typically, the one that used to take the longest.

Reducing the time to build these environments to minutes requires more than just infrastructure. It requires a platform that can support the deployment of fully configured environments. Paradigms like patterns and clones, discussed earlier, are essential to achieve that speed of deployment.

Additionally, these paradigms help with reuse of expertise within the organization as well. Environments capture the expertise needed to deploy various aspects of the application, be it infrastructure (IT team), presentation layer (UI team), business logic (app server team), database layer (DBAs). All the tester has to do is click a button or call an API to deploy the environment. The individual components are automatically deployed based on the definition of each component captured in the environment definition.

6. How does this impact running an automated test suite?

Automated testing at each stage can be made more efficient and predictable by using consistent, fully configured environments that can be deployed using API calls. Continuous Integration and Continuous Delivery can actually be realized if the environments of desired size and configuration can be deployed within seconds and minutes versus hours and days.

7. Getting the right data to test with, or setting up the right data is also a problem area for many teams. What strategies or tools help with this?

A few strategies can help with test data management:

  • Populating data in base environments which can then be cloned into new copies of the environment when needed. Once testing is done, the environment along and data can be thrown away. It is cheap to recreate a new environment with the desired data set.
  • Creating a database as part of environment creation. In this model, the data is created by using some automation code. This slows the set-up of the environments but ensures that the data creation is based on the latest desired state of the test data set.
  • Using data virtualization and data-masking tools to continuously keep a close-to-production copy of test data for testing. This is especially important in the later stages of testing like pre-prod certification testing.

8. Are cloud-based environments also a popular choice for those doing continuous integration and or continuous testing?

Absolutely. The key requirement for CI/CD—speed of execution and feedback—cannot be achieved until test environments can be deployed on-demand, in an automated fashion, in a consistent state, and with desired complexity.

9. We know containers are a stripped down wrapper around software components or services. How does Skytap use containers?

Skytap views containers as another component of the application stack just like all other components: infrastructure, middleware and applications. As customers modernize parts of their application that are a good fit for container technology, they are able to use them in Skytap environments, along with the other parts of their application that are not containerized. This is a very common scenario for traditional enterprise applications that tend to incorporate new technologies and are modernized over time.

With containers in Skytap, not only are customers able to use the container toolchain with container hosts in Skytap, but they are also able to still achieve the same efficiencies in their SDLC that they have traditionally done with environments. Additionally, Skytap customers have full control over their container hosts. This is not available in many container services, but is often desired by our enterprise customers. Customers can run containers on host VMs in their Skytap environments. They can also use the Skytap driver for Docker Machine to manage containers running on hosts in Skytap. If you are interested in learning more about Skytap’s support for containers, please contact

10. How do you see the use of containers impacting testing?

Container technology can affect testing in many ways. Some of these are:

  • Deploying test environments: Application test environments that are entirely container-based or those that use containers for some application components require new ways for deploying the test environments while still meeting the requirements of consistency.
  • Testing both code and deployment: Containers encapsulate both the code changes and the deployment process by design. Containers deploy fully configured application components across the testing pipeline. Testing that traditionally only used to focus on code changes and deployment changes in isolation will need to accommodate both.
  • Integration with tools across the testing toolchain: Consideration must be given to the choice of build, testing, and automation tools that can work with traditional application components side-by-side with container-based components.
Sumit Mehrotra
Sumit Mehrotra is Sr. Director of Product Management at Skytap, a role in which he is responsible for product strategy and roadmaps. Prior to Skytap, Sumit worked at Microsoft in different roles and has shipped a number of products, including Windows Azure and Windows operating system. Sumit holds an MBA from University of Chicago Booth School of Business and a Masters in Computer Science from Boston University.

The Related Post

Special considerations that should be applied to an application running in the cloud. Over the last weeks, I have found myself in several rather intense discussions about “cloud testing”: what it is, what it isn’t, and what it means for testing and QA professionals. The major source of confusion in these discussions usually revolves around ...
Run your TestArchitect API and headless browser tests inside Docker containers as easy as flipping a switch Docker is a virtualization platform enabling you to create containers – mini virtual machines— which have their own predefined environment, including file system, libraries and settings. Best of all, these light-weight images eat up only a few megabytes ...
Disclaimer: This article was originally published in the LogiGear Magazine, but has been updated for the June 2020 issue. Leading an organization into Continuous Delivery is a daunting task that usually takes software development teams months if not years. This guide covers the main points that organizations need to consider to increase their chance of ...
Cloud computing has been the buzzword in the world of Information Technology for quite some time and it is likely to retain that status in the coming years. Cloud computing has been helping business enterprises deliver services faster and cheaper compared to all other existing delivery models. Small and medium business enterprises have changed their ...
From adopting the culture, to implementing Continuous Delivery With the relative newness of DevOps, there are not yet a ton of DevOps books. That’s why we’ve assembled a list of the 7 best DevOps books based on four criteria: the number of ratings from Amazon, the average Amazon rating, number of ratings from GoodReads and the ...
A test team’s job is to report test results, not set or guarantee that you will meet the SLAs. In the rush to cloud services, with everything-as-a-service, you will hear people talking about SLAs. What is this about and what does it have to do with testing? A Service Level Agreement, or SLA, is a ...
LogiGear Magazine – February 2013 – The Rapidly Changing Software Testing Landscape
Having the right skills and experience, even if you have to go outside, is essential for designing tests for large-scale cloud deployments. Moving existing applications to a cloud environment adds new dimensions to testing. One of the primary reasons for moving to the cloud is scalability. Capacity to handle traffic and data transfer can be ...
DevOps for Test Teams By Michael Hackett Now that Dev Teams have had a little time to settle into the Agile, the new wave of process optimization has arrived. DevOps. DevOps has been described as Agile on Steroids. DevOps has also been described as Agile for Operations/IT. I like both of those descriptions as well ...
LogiGear University announces the launch of a new, free video series on Testing in DevOps and Continuous Testing which is available today.
The book is an incredibly effective and valuable guide that details the risks that arise when deploying cloud solutions. More importantly, it provides details on how to test cloud services, to ensure that the proposed cloud service will work as described. It is a great start to the topic. The 6 chapters detail a paradigm ...
Do you want to speed up your automated tests by a factor of 10 and deploy your application continuously? In this article we share how the JIRA development team at Atlassian has accomplished this using Stages in Bamboo. Stages have allowed the JIRA Development team to take a week’s worth of testing and condense it ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news