What Exactly Is ‘Cloud Testing’ Anyway?

Special considerations that should be applied to an application running in the cloud.

Over the last weeks, I have found myself in several rather intense discussions about “cloud testing”: what it is, what it isn’t, and what it means for testing and QA professionals. The major source of confusion in these discussions usually revolves around the definition of cloud itself; if you try look up cloud computing on the Internet, you will find it hard to get a formal definition. Wikipedia says it outright: “Cloud computing is a jargon term without a commonly accepted non-ambiguous scientific or technical definition.” Duh!

Since the primary goal of this article is to talk about cloud applications and testing thereof, I’ve tried to distill a number of common traits for “cloud applications” based on the above and similar articles (let me know if you disagree!).

Common for “cloud applications” is that they:

  • Run on a virtualized hardware and a software stack that can be moved and replicated between physical machines as needed.
  • Share common physical resources with other cloud applications (disk, network, data stores, etc.).
  • Are built to be highly scalable in real-time – meaning that they can handle high increases in load by dynamically scaling to more physical resources as needed.
  • Are predominately accessed using standard network protocols.
  • Use HTML and other web technologies for providing both their front-end and management UIs.
  • Provide APIs for integration and management – possibly made available to users or third-party application vendors.
  • Consume third-party APIs for providing common services and functionality – things like authentication (OpenID, Facebook), storage (DropBox, Google Drive), messaging (Twitter, Gmail), geo-functionality (GooglMaps) etc.
  • Tend to use NoSQL data stores – primarily for managing large amounts of unstructured data.

The “cloud” itself comes down to being the infrastructure that hosts a “cloud application”; it is usually either public (Amazon, Rackspace, etc.), private, or a combination of the two – and can offer many different levels of service (IaaS, PaaS, SaaS, etc.).

So, given these basic characteristics, what should testers be thinking of when tasked with testing a “cloud application” – or more likely, a web application or API that is running “in the cloud”? Are there any specific traits related to a cloud application that mandate extra consideration as opposed to if the application is deployed on an old and dusty server in the corner of your office (running Windows 2000?).

My immediate answer to this question used to be “No.” A web application or API needs to be tested in the same way no matter how it is deployed. It still has to work and perform as required, and testing is no different for different deployment scenarios. Or so I thought – thanks to my colleagues’ persistence, I have started to open up to a more nuanced answer; perhaps there are a number of aspects that a tester needs to give special consideration to when tasked with testing a “cloud application” – many of which are related to the infrastructural nature of the cloud. For example:

  • Performance – Applications running in a cloud run on hardware that you might not have any control over, and that they share with other applications. Therefore, ensuring both performance and required scalability is extremely important. Be sure to test performance in a cloud environment similar to the one you will be using in production. If you know that your application shares resources with other applications under your control, run load tests on both at the same time to see if they affect each other. In production, be sure to use monitoring as a means to continuously validate both performance and functionality while your application is in production – ensuring that it scales as required. Using cloud resources to scale under load can be costly, so knowing where that breakpoint is and monitoring to see how close you are to it can also help you budget correctly for your infrastructure needs.
  • Security – Since cloud applications usually share resources and infrastructure with others, you have to give extra consideration to data privacy and access control issues. Is sensitive data encrypted when stored? Are access control mechanisms in place in all possible situations and at all levels? This is just as valid for applications hosted in a private cloud; data intrusion and “theft” could even happen “by accident” if, for example, a backup for one cloud application happens to access resources or data related to another application.
  • Third-party dependencies – Cloud applications are likely to consume external APIs and services for providing some of their functionality. You should consider testing and monitoring these as if they were part of your own solution (since that’s what they are from your users’ point of view). You want to make sure they work as you need them to and you want to be the first to know when they don’t.

One common interpretation of “cloud testing” that many vendors seem to adhere to is using the cloud to run or manage the tests themselves. For example, testers can use the cloud to generate massive distributed load tests, simulate a large number of mobile devices, or run functional and performance monitors from all over the world. Although these are all extremely valuable offerings themselves, they are not very specific for testing cloud applications. So, calling it “cloud testing” is kind of a stretch in some situations.

So, did this initial insight help me in understanding what “cloud testing” is? Well, to be honest, not really. Although I do agree that there are things to be aware of when testing an application in the cloud, I still struggle with “cloud testing” being something separate than “regular” performance, integration or security testing.

All of these just need some special consideration and understanding when applied to an application running in the cloud.

Ole Lensmar
Ole Lensmar is Chief Architect and co-founder of SmartBear Software in Sweden, formerly the maker of SoapUI, Eviware Software, acquired by SmartBear in 2011. He also co-founded base8, an XML oriented consulting company in 1996, acquired by the publicly traded Mogul in 1998. Ole worked as CTO, product owner and lead evangelist for Mogul’s software portfolio, including an XML based CMS and a high performance search engine. With Niclas Reimertz, Ole created SoapUI, now the most used open source testing tool in the world with five million downloads and one million active users. Ole’s astute eye for technology trends and the test community led to his promotion to chief architect at SmartBear. Living his passion for software development, he writes a weekly column for Network World about software quality and blogs in general about coding, quality and all other fun things in life.

The Related Post

LogiGear Magazine – December 2013 – Cloud Testing
In this article, I share some of my experiences and observations on training teams, mainly in corporate settings. The operative word here is “team”, not “individual”. When training teams or groups in an organization, many of the considerations and benefits are different than those for the individual. We’ll examine those differences and I will share successful solutions. ...
Throw away clunky hyper-visors, and stop thinking about computer hardware and software license during your development projects. The first thing you think about when you hear “The Cloud” may not be development and testing. The Cloudy market is filled with SaaS applications, hosting, and cloud-based file systems. All are very useful, and offer a clear ...
Fitting QA into a modern DevOps group In a traditional software engineering organization, the QA group is often seen as separate from the Development group. Developers and testers have different roles, different responsibilities, different job descriptions, and different management. They are two distinct entities. However, for folks outside the engineering team – say in Operations ...
The book is an incredibly effective and valuable guide that details the risks that arise when deploying cloud solutions. More importantly, it provides details on how to test cloud services, to ensure that the proposed cloud service will work as described. It is a great start to the topic. The 6 chapters detail a paradigm ...
Sauce Lab’s Perspective on Integration with Continuous Testing The term ‘DevOps’ implies that implementing an effective development and production workflow is as easy as developing the collaborative interaction between developers and IT operations. On the surface, DevOps may appear as a simple and straightforward idea but, as you will find out, there is more than meets ...
Continuous Testing and Continuous Monitoring What is the goal of Continuous Integration? Is it to enable Continuous Delivery of the code developers’ produce out to users? Yes, eventually. But first and foremost it is to enable ongoing test and verification of the code. It is to validate that the code produced and integrated with that ...
DevOps may be the next big buzzword, but Test teams really need to focus on its little sister, Continuous Delivery If you pay attention to trends in software development—from the perspective of what some sophisticated teams are doing, what articles and books are being written, to conference topics, you may have noticed the tools being ...
Disclaimer: This article was originally published in the LogiGear Magazine, but has been updated for the June 2020 issue. Leading an organization into Continuous Delivery is a daunting task that usually takes software development teams months if not years. This guide covers the main points that organizations need to consider to increase their chance of ...
How to Adopt the “Third Way” in the Dojo’s Method to Master CD In The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations the authors describe “The Three Ways” – the underlying principles forming the basis for all DevOps practices. 
For this month’s book review, I read Continuous Testing for DevOps Professionals: A Practical Guide from Industry Experts, by various authors and edited by Eran Kinsbruner. The book is divided into 4 sections: Fundamentals of Continuous Testing, Continuous Testing for Web Apps, Continuous Testing for Mobile Apps, and The Future of Continuous Testing. The Fundamentals ...
From the culture shift, to differences in Agile, Dave Farley and Michael Hackett discuss the nitty gritty of Testing in DevOps. For this issue of LogiGear Magazine, our very own Michael Hackett sat down with one of the godfathers of Continuous Delivery, David Farley. In this exclusive interview, David discusses how test teams and automation ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe