Test in Production?

This post is part of the Pride & Paradev series

13691-NPAE1I

With continuous deployment, it is common to release new software into production multiple times a day. A regression test suite, no matter how well designed, may still take over 10 minutes to run, which can lead to bottlenecks in releasing changes to production.

So, do you even need to test before going live? Why not just test changes in production?

Test changes in production

The website for The Guardian, the UK’s third largest newspaper, deploys on average 11 times a day, of which all changes are tested in production.

Once the code is in production, QA can really start.” “Sometimes deployments go wrong. We expect that; and we accept it, because people (and machines) go wrong. But the key to dealing with these kind of mistakes is not to lock down the process or extend the breadth, depth and length of regression tests. The solution is to enable people to fix their mistakes quickly, learn, and get back to creating value as soon as possible.”

~ Andy Hume on Real-time QA [The Guardian Developer Blog]

The key to testing changes as soon as they hit production is to have real time, continuous real user experience monitoring. This includes metrics like page views and page load time, which directly correlate to advertising revenue, an incentive to keep these healthy.

More comprehensive automated acceptance tests can be written in a non-destructive style that means they can be run in production. This means that these can be run immediately following a fresh production deployment, and as feedback about the tests is received, any issues can be remedied immediately into production and tested again.

Test changes before production

There are not many businesses that are able to release software without any form of testing into production: whether there be legislative requirements requiring testing, or the risk of introducing errors is too high for its target market.

Whilst automated regression tests do take longer to run than unit or integration tests, there are ways to manage these to ensure the quickest path into production. These strategies include running tests in parallel, only running business critical tests, only running against the single most popular browser, or only running tests that are directly related to your changes.

You can set up a deployment pipeline that runs a selected subset of tests before deploying into production then running the remaining tests (in a test environment). Any of the issues found in subsequent tests are judged to see whether they warrant another immediate release or whether they can be included in the next set of changes being deployed into production.

Whilst you definitely should run tests before deploying to production, it doesn’t mean that this has to drastically hinder your ability to continuously deploy.

Alister

Alister is an Excellence Wrangler for WordPress.com at Automatic. He has extensive experience in automated software testing and establishing quality engineering cultures in lean cross-functional software development teams. He lives in Brisbane, Australia with his wife and three sons, and writes a popular software testing blog at watirmelon.com.

Alister Scott
Alister is an Excellence Wrangler for Wordpress.com at Automatic. He has extensive experience in automated software testing and establishing quality engineering cultures in lean cross-functional software development teams. He lives in Brisbane, Australia with his wife and three sons, and writes a popular software testing blog at http://watirmelon.com.

The Related Post

How to ensure a successful test-driven environment In order to ensure a higher quality product is released in the end, many teams have turned to test-driven development. Under this scenario, quality assurance metrics professionals first create various QA tests, and then software engineers code based on these tests, typically while using a robust enterprise test management ...
LogiGear University announces the launch of a new, free video series on Testing in DevOps and Continuous Testing which is available today.
Cloud computing has been the buzzword in the world of Information Technology for quite some time and it is likely to retain that status in the coming years. Cloud computing has been helping business enterprises deliver services faster and cheaper compared to all other existing delivery models. Small and medium business enterprises have changed their ...
Aligning the Dev and Ops Teams DevOps as a philosophy has had as its centerpiece the principle that Dev and Ops teams need to align better. This is a people and organizational principle, not a process centric principle. To me this is more important when adopting DevOps than any other capability or tool. My last post ...
DevOps for Test Teams By Michael Hackett Now that Dev Teams have had a little time to settle into the Agile, the new wave of process optimization has arrived. DevOps. DevOps has been described as Agile on Steroids. DevOps has also been described as Agile for Operations/IT. I like both of those descriptions as well ...
LogiGear Magazine June Testing in Continuous Delivery Issue 2017
DevOps has been described as Agile on Steroids; DevOps has also been described as Agile for Operations/IT. I like both of those descriptions. Many organizations want Development, Test, and Operations teams to move to DevOps now. DevOps is a big topic, but DevOps is not the focus of this article. We will not be talking ...
Fitting QA into a modern DevOps group In a traditional software engineering organization, the QA group is often seen as separate from the Development group. Developers and testers have different roles, different responsibilities, different job descriptions, and different management. They are two distinct entities. However, for folks outside the engineering team – say in Operations ...
Having the right skills and experience, even if you have to go outside, is essential for designing tests for large-scale cloud deployments. Moving existing applications to a cloud environment adds new dimensions to testing. One of the primary reasons for moving to the cloud is scalability. Capacity to handle traffic and data transfer can be ...
Continuous Testing and Continuous Monitoring What is the goal of Continuous Integration? Is it to enable Continuous Delivery of the code developers’ produce out to users? Yes, eventually. But first and foremost it is to enable ongoing test and verification of the code. It is to validate that the code produced and integrated with that ...
LogiGear Magazine – December 2013 – Cloud Testing
…On what you need to know before making the transition to EaaS 1. What are the main differences between cloud-based environments and cloud infrastructure? An environment is a collection of infrastructure elements working in conjunction to enable an application stack to work. For example, a simple 3-tier application, with a web front-end component, a business logic ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe