Test in Production?

This post is part of the Pride & Paradev series

13691-NPAE1I

With continuous deployment, it is common to release new software into production multiple times a day. A regression test suite, no matter how well designed, may still take over 10 minutes to run, which can lead to bottlenecks in releasing changes to production.

So, do you even need to test before going live? Why not just test changes in production?

Test changes in production

The website for The Guardian, the UK’s third largest newspaper, deploys on average 11 times a day, of which all changes are tested in production.

Once the code is in production, QA can really start.” “Sometimes deployments go wrong. We expect that; and we accept it, because people (and machines) go wrong. But the key to dealing with these kind of mistakes is not to lock down the process or extend the breadth, depth and length of regression tests. The solution is to enable people to fix their mistakes quickly, learn, and get back to creating value as soon as possible.”

~ Andy Hume on Real-time QA [The Guardian Developer Blog]

The key to testing changes as soon as they hit production is to have real time, continuous real user experience monitoring. This includes metrics like page views and page load time, which directly correlate to advertising revenue, an incentive to keep these healthy.

More comprehensive automated acceptance tests can be written in a non-destructive style that means they can be run in production. This means that these can be run immediately following a fresh production deployment, and as feedback about the tests is received, any issues can be remedied immediately into production and tested again.

Test changes before production

There are not many businesses that are able to release software without any form of testing into production: whether there be legislative requirements requiring testing, or the risk of introducing errors is too high for its target market.

Whilst automated regression tests do take longer to run than unit or integration tests, there are ways to manage these to ensure the quickest path into production. These strategies include running tests in parallel, only running business critical tests, only running against the single most popular browser, or only running tests that are directly related to your changes.

You can set up a deployment pipeline that runs a selected subset of tests before deploying into production then running the remaining tests (in a test environment). Any of the issues found in subsequent tests are judged to see whether they warrant another immediate release or whether they can be included in the next set of changes being deployed into production.

Whilst you definitely should run tests before deploying to production, it doesn’t mean that this has to drastically hinder your ability to continuously deploy.

Alister

Alister is an Excellence Wrangler for WordPress.com at Automatic. He has extensive experience in automated software testing and establishing quality engineering cultures in lean cross-functional software development teams. He lives in Brisbane, Australia with his wife and three sons, and writes a popular software testing blog at watirmelon.com.

Alister Scott
Alister is an Excellence Wrangler for Wordpress.com at Automatic. He has extensive experience in automated software testing and establishing quality engineering cultures in lean cross-functional software development teams. He lives in Brisbane, Australia with his wife and three sons, and writes a popular software testing blog at http://watirmelon.com.

The Related Post

  LogiGear_Magazine_June 2016_Testing in the New World of DevOps  
Special considerations that should be applied to an application running in the cloud. Over the last weeks, I have found myself in several rather intense discussions about “cloud testing”: what it is, what it isn’t, and what it means for testing and QA professionals. The major source of confusion in these discussions usually revolves around ...
LogiGear University announces the launch of a new, free video series on Testing in DevOps and Continuous Testing which is available today.
Throw away clunky hyper-visors, and stop thinking about computer hardware and software license during your development projects. The first thing you think about when you hear “The Cloud” may not be development and testing. The Cloudy market is filled with SaaS applications, hosting, and cloud-based file systems. All are very useful, and offer a clear ...
DevOps for Test Teams By Michael Hackett Now that Dev Teams have had a little time to settle into the Agile, the new wave of process optimization has arrived. DevOps. DevOps has been described as Agile on Steroids. DevOps has also been described as Agile for Operations/IT. I like both of those descriptions as well ...
Run your TestArchitect API and headless browser tests inside Docker containers as easy as flipping a switch Docker is a virtualization platform enabling you to create containers – mini virtual machines— which have their own predefined environment, including file system, libraries and settings. Best of all, these light-weight images eat up only a few megabytes ...
…On what you need to know before making the transition to EaaS 1. What are the main differences between cloud-based environments and cloud infrastructure? An environment is a collection of infrastructure elements working in conjunction to enable an application stack to work. For example, a simple 3-tier application, with a web front-end component, a business logic ...
Disclaimer: This article was originally published in the LogiGear Magazine, but has been updated for the June 2020 issue. Leading an organization into Continuous Delivery is a daunting task that usually takes software development teams months if not years. This guide covers the main points that organizations need to consider to increase their chance of ...
Introduction Everything changes. It’s the only constant. The landscape of software testing is undergoing a fast and dramatic change driven by societal, business and technology changes that have placed software everywhere. Computing is ubiquitous. There is hardly anything today that doesn’t contain a CPU or information transmission capability, with software to drive it. From smart toasters ...
LogiGear Magazine – December 2013 – Cloud Testing
By Jez Humble and David Farley Continuous Delivery from Jez Humble and David Farley is an important contribution to the field of software development. It takes continuous integration to the logical conclusion and covers how to set up a continuous integration system, delving into everything from check-in to delivery to production. It doesn’t state you ...
LogiGear Magazine June Testing in Continuous Delivery Issue 2017

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe