An Interview with Skytap’s Sumit Mehrotra

…On what you need to know before making the transition to EaaS

LogiGear_Skytap's Sumit

1. What are the main differences between cloud-based environments and cloud infrastructure?

An environment is a collection of infrastructure elements working in conjunction to enable an application stack to work.

For example, a simple 3-tier application, with a web front-end component, a business logic component and a database component requires, at a minimum, the following infrastructure components: a few VMs, a few storage disks, and a network. All of these components are required, and they need to work together as an environment for the application to be functional. Note, that as the complexity of the application stack grows, the definition of an environment grows bigger than just infrastructure components. It can include VPNs, public IP addresses, an object store, and other service end-points.

From an application development and testing perspective, the application environment is the starting point of productive work. However, we see teams spending a large amount of time just getting the infrastructure components to a working state and hence losing out on a good portion of their productive work. Working with environments as a unit of work can enable these teams to be more productive and hence service customer and business requirements at a faster pace with higher quality.

2. What is “environment or infrastructure as code?” How can we use this idea in testing?

As the name suggests, infrastructure-as-Code is practice of representing infrastructure as code blocks so that they can be executed whenever needed to build an application stack (environment) in a repeatable and consistent manner. Depending on the actual tool used, it can allow you a variety of things including:

  • Deploy basic infrastructure elements, VMs, disks, networks, etc.
  • Install and configure operating systems
  • Deploy and configure application components
  • Populate application data

With a repeatable and consistent process, bugs can be reproduced, fixed, and validated efficiently. Developers can recreate the same environment that a tester is using to find bugs by using the same Infrastructure-as-Code scripts as the tester is using. This practice goes a long way in solving the problem of ‘it-works-on-my-machine’ syndrome.

There are various paradigms available to create consistent environments in a repeatable manner. Two such popular paradigms are “patterns” and “clones.” With patterns, you can create the environment from scratch each time—allowing you to progress the same exact code at each point of your software delivery pipeline. But it takes time for the environment to be provisioned and can be error-prone especially, for very complex environments. With clones, you can create another copy of the environment from a pre-existing copy. The process can be very fast, even for complex environments. However, moving the same clone through your SDLC can be a heavyweight process.

There is no right or wrong paradigm here. Successful teams have chosen a combination of paradigms and have made trade-offs between time and tooling consistency, to achieve the velocity they need in their software delivery lifecycle.

3. How do companies use cloud-based environments to facilitate testing in DevOps?

Testing especially in the DevOps world has the following key requirements:

  • Instant self-service access to application stacks for testing
  • Support the complexity and scale of the application stack needed at various stages of testing
  • Consistent and reproducible ways to create test environments
  • Ability to collaborate and share environments with developers and testers as part of a feedback loop. Constant feedback is a key aspect of Agile development
  • Ability to ensure that the right resources are available at any time and are being utilized judiciously

All of these requirements are hard to meet without a cloud-based platform. So, cloud-based environments are essential to facilitate testing workflows.

4We know that getting the right environments for testing can be difficult, and sometimes it’s out of a Test team’s control. How are innovative organizations solving this challenge?


Organizations are increasingly turning to the cloud for on-demand and self-service access test environments, but not every cloud platform is suited for the needs of the applications in an organization’s portfolio. Innovative organizations evaluate the environment needs of their applications, and some of the criteria that we have seen customers use are as follows:

  • Configurability – Can the platform deliver test environments that capture the complexity of my application at each stage of testing?
  • Consistency – Can the platform deliver test environments that are in the exact state that we need them to be, whenever I want?
  • Collaboration – Can the platform make it easier for my team (Devs + QA + Ops) to work together more productively?
  • Control – Does the platform have controls to ensure the right people have the appropriate resources to do their jobs?

5. How long does it take to build these environments? Some Test teams do not have dedicated IT members—does the Test team often do this?

The time to get a fully configured environment to test can vary in range from a few minutes to a few weeks. The availability of cloud platforms, and especially IaaS has reduced the far-end of the range to a few days primarily by taking the acquisition of hardware out of the equation. This is the lowest layer of the environment and typically, the one that used to take the longest.

Reducing the time to build these environments to minutes requires more than just infrastructure. It requires a platform that can support the deployment of fully configured environments. Paradigms like patterns and clones, discussed earlier, are essential to achieve that speed of deployment.

Additionally, these paradigms help with reuse of expertise within the organization as well. Environments capture the expertise needed to deploy various aspects of the application, be it infrastructure (IT team), presentation layer (UI team), business logic (app server team), database layer (DBAs). All the tester has to do is click a button or call an API to deploy the environment. The individual components are automatically deployed based on the definition of each component captured in the environment definition.

6. How does this impact running an automated test suite?

Automated testing at each stage can be made more efficient and predictable by using consistent, fully configured environments that can be deployed using API calls. Continuous Integration and Continuous Delivery can actually be realized if the environments of desired size and configuration can be deployed within seconds and minutes versus hours and days.

7. Getting the right data to test with, or setting up the right data is also a problem area for many teams. What strategies or tools help with this?

A few strategies can help with test data management:

  • Populating data in base environments which can then be cloned into new copies of the environment when needed. Once testing is done, the environment along and data can be thrown away. It is cheap to recreate a new environment with the desired data set.
  • Creating a database as part of environment creation. In this model, the data is created by using some automation code. This slows the set-up of the environments but ensures that the data creation is based on the latest desired state of the test data set.
  • Using data virtualization and data-masking tools to continuously keep a close-to-production copy of test data for testing. This is especially important in the later stages of testing like pre-prod certification testing.

8. Are cloud-based environments also a popular choice for those doing continuous integration and or continuous testing?

Absolutely. The key requirement for CI/CD—speed of execution and feedback—cannot be achieved until test environments can be deployed on-demand, in an automated fashion, in a consistent state, and with desired complexity.

9. We know containers are a stripped down wrapper around software components or services. How does Skytap use containers?

Skytap views containers as another component of the application stack just like all other components: infrastructure, middleware and applications. As customers modernize parts of their application that are a good fit for container technology, they are able to use them in Skytap environments, along with the other parts of their application that are not containerized. This is a very common scenario for traditional enterprise applications that tend to incorporate new technologies and are modernized over time.

With containers in Skytap, not only are customers able to use the container toolchain with container hosts in Skytap, but they are also able to still achieve the same efficiencies in their SDLC that they have traditionally done with environments. Additionally, Skytap customers have full control over their container hosts. This is not available in many container services, but is often desired by our enterprise customers. Customers can run containers on host VMs in their Skytap environments. They can also use the Skytap driver for Docker Machine to manage containers running on hosts in Skytap. If you are interested in learning more about Skytap’s support for containers, please contact

10. How do you see the use of containers impacting testing?

Container technology can affect testing in many ways. Some of these are:

  • Deploying test environments: Application test environments that are entirely container-based or those that use containers for some application components require new ways for deploying the test environments while still meeting the requirements of consistency.
  • Testing both code and deployment: Containers encapsulate both the code changes and the deployment process by design. Containers deploy fully configured application components across the testing pipeline. Testing that traditionally only used to focus on code changes and deployment changes in isolation will need to accommodate both.
  • Integration with tools across the testing toolchain: Consideration must be given to the choice of build, testing, and automation tools that can work with traditional application components side-by-side with container-based components.
Sumit Mehrotra
Sumit Mehrotra is Sr. Director of Product Management at Skytap, a role in which he is responsible for product strategy and roadmaps. Prior to Skytap, Sumit worked at Microsoft in different roles and has shipped a number of products, including Windows Azure and Windows operating system. Sumit holds an MBA from University of Chicago Booth School of Business and a Masters in Computer Science from Boston University.

The Related Post

This post is part of the Pride & Paradev series With continuous deployment, it is common to release new software into production multiple times a day. A regression test suite, no matter how well designed, may still take over 10 minutes to run, which can lead to bottlenecks in releasing changes to production. So, do ...
Aligning the Dev and Ops Teams DevOps as a philosophy has had as its centerpiece the principle that Dev and Ops teams need to align better. This is a people and organizational principle, not a process centric principle. To me this is more important when adopting DevOps than any other capability or tool. My last post ...
Introduction Everything changes. It’s the only constant. The landscape of software testing is undergoing a fast and dramatic change driven by societal, business and technology changes that have placed software everywhere. Computing is ubiquitous. There is hardly anything today that doesn’t contain a CPU or information transmission capability, with software to drive it. From smart toasters ...
Continuous Testing and Continuous Monitoring What is the goal of Continuous Integration? Is it to enable Continuous Delivery of the code developers’ produce out to users? Yes, eventually. But first and foremost it is to enable ongoing test and verification of the code. It is to validate that the code produced and integrated with that ...
Fitting QA into a modern DevOps group In a traditional software engineering organization, the QA group is often seen as separate from the Development group. Developers and testers have different roles, different responsibilities, different job descriptions, and different management. They are two distinct entities. However, for folks outside the engineering team – say in Operations ...
LogiGear Magazine – December 2013 – Cloud Testing
It’s no secret that the cloud is growing at an exponential rate. By 2016, two-thirds of the world’s server workloads will exist in the cloud. But according to Cisco’s 2012 Cloud Index, less than half of server workloads currently run in the cloud. Closing the gap between current capabilities and future requirements is a mission-critical ...
In this article, I share some of my experiences and observations on training teams, mainly in corporate settings. The operative word here is “team”, not “individual”. When training teams or groups in an organization, many of the considerations and benefits are different than those for the individual. We’ll examine those differences and I will share successful solutions. ...
LogiGear Magazine June Issue 2018: TESTING in DEVOPS
The book is an incredibly effective and valuable guide that details the risks that arise when deploying cloud solutions. More importantly, it provides details on how to test cloud services, to ensure that the proposed cloud service will work as described. It is a great start to the topic. The 6 chapters detail a paradigm ...
DevOps for Test Teams By Michael Hackett Now that Dev Teams have had a little time to settle into the Agile, the new wave of process optimization has arrived. DevOps. DevOps has been described as Agile on Steroids. DevOps has also been described as Agile for Operations/IT. I like both of those descriptions as well ...
LogiGear Magazine – February 2013 – The Rapidly Changing Software Testing Landscape

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news