Continuous Delivery: Everything You Need to Know

Disclaimer: This article was originally published in the LogiGear Magazine, but has been updated for the June 2020 issue.

Leading an organization into Continuous Delivery is a daunting task that usually takes software development teams months if not years. This guide covers the main points that organizations need to consider to increase their chance of success when adopting Continuous Delivery. Topics to be covered include:

The fundamental underpinnings of CD are dependent on 2 parts: culture change and commitments to the toolchain. Typically, there is more emphasis on commitment to the toolchain; this is because CD requires an increase in Automation–both Task Automation and Test Automation. Each section contains everything businesses need to consider when it comes to building and implementing a successful Continuous Testing program, from the toolchain to best practices, developing a strategy, and discussing options on how to get started.

Continuous Integration vs. Continuous Delivery

First, let’s establish the definitions of Continuous Delivery and Continuous Integration and establish the differences. Continuous Integration is a software engineering practice in which the changes made by developers to working copies of code are added to the mainline code base on a frequent basis, and immediately tested. The goal is to provide rapid feedback so that if a defect is introduced into the mainline, it can be identified quickly and corrected as soon as possible. 

Continuous Integration software tools are often used to automate testing and build a document trail. Because CI detects deficiencies early on in development, defects are typically smaller, less complex, and easier to resolve. In the end, well-implemented CI reduces the cost of software development and reduces the time to market.

Contrasted, Continuous Delivery is a “Software Engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software faster and more frequently. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for Continuous Delivery.”

The main difference is that CI focuses on Dev and Test teams, whereas CD focuses on Ops teams. Where CI ends, CD picks up: promoting builds, versions, and deployments up closer to and finally, the live/production environments.

Continuous Delivery Pipeline

Continuous Delivery has been related to a production pipeline. The main idea is that the “Continuous Delivery pipeline” is the image and description of product flow through the development process and toolchain. This pipeline flow is a variation of a manufacturing production line. 

It became very clear in Agile that quality is everyone’s job. Likewise, everyone owns a piece of quality. In CD, everyone owns deployment and everyone shares a piece of deployment. Many things must change about the product to get these ideas implemented: architectural changes, tool changes, learning new skills to facilitate culture changes and rethinking the idea of release

As with all production lines, there will be bottlenecks. The goal with pipelines is a constant flow of high quality and finished/deployment-ready work. Keep the flow going. Keep the pipeline moving. Get lots of immediate feedback to the team.

The Continuous Delivery pipeline is a huge topic within itself, and it could be written about extensively. In fact, Michael Hackett already has! Check out Michael’s article, Mega Trends in Testing: Continuous Delivery, Production Line, and the Deployment Pipeline, to get a full, in-depth understanding of the CD pipeline.

Continuous Delivery Tools

The financial commitments to the toolchain often require a large upfront investment for organizations, this can be hard due to the necessity of increasing task Automation. Tools will be needed for the following areas:

  • Cloud/EaaS, Iaas, Paas
  • Configuration Management (e.g. Chef and Puppet)
  • Containers (e.g. Docker and Packer)
  • CI/Version Control (e.g. Jenkins, Bamboo, TeamCity, and Buildbot)
  • Deployment (e.g. XebiaLabs Deploy, Go Thoughtworks, IBM UrbanCode Deploy, and Code Stream)
  • Monitoring (e.g. New Relic and App Dynamics)
  • Source Control (e.g. Git SVN)
  • Test Automation (e.g. Selenium, TestArchitect, Cucumber, and jBehave)

As you can see, for each different type of tool, there are many different options. At the basic level, they all perform the same task, but which one is right for you depends on your organization’s needs and processes, as well as your Automation Tool of choice. Make sure that you do proper research into prospective tools before beginning your CI/CD implementation.

Continuous Delivery Testing / Testing in Continuous Delivery

The figure below illustrates all the places where Test Automation appears in the Continuous Delivery Pipeline.

In Continuous Delivery, Testing appears at every stage of the pipeline–it is no longer isolated. It is important to note that test teams, including Test Automation, cannot become the bottleneck in the Continuous Delivery pipeline. The main reason is that if Software Testing or Test Automation does become a bottleneck, it will be ignored and quality will certainly deteriorate.

For the product pipeline to keep moving with immediate feedback, you don’t have to take one Automated UI regression suite and have it run again, and again, and again. That doesn’t ensure quality or confidence. There are certain times, particularly with newly committed code, that unit tests are the most efficient for immediate feedback. There are other times where API or service level testing are the best strategies, and still, other places where the tests best inserted into the pipeline are UI-driven tests. Testing in Continuous Delivery comes with many perks including higher efficiency and higher ROI, but it’s not an easy implementation; there are many aspects that can’t be covered in one magazine article, and that’s why we wrote our eBook Testing in Continuous Delivery. It features in-depth industry knowledge that leader’s need to know in order to successfully lead their team into Continuous Delivery.

Continuous Delivery Test Strategy

What is commonly referred to as a “Continuous Delivery Test Strategy” is in actuality Continuous Testing? Continuous Testing (CT) is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate, and CT has its own Automation strategy. It is important to remember that as you build test suites for the CD pipeline, Test Automation does not show a lack of bugs. Test Automation shows consistency with the last iteration; it shows stability and predictability. Not finding bugs does not indicate that there are no bugs-it indicates that the bugs that haven’t been found yet are still hidden.

The test strategy for Continuous Testing is a full reexamination of what gets tested where. This with the Lean Software development (LSD) idea of “quality at every step” is the foundation of a CD test strategy. It involves reexamining, what you have been calling, “regression” and developing multiple lean and mean suites for immediate feedback.

It also means rethinking what and how you automate. Old style Automation-even small isolated automated tests need to be reexamined and replaced by efficient, low-level tests as well as longer customer-focused transaction workflows. Rethinking regression and Automation is a big task for even the most effective teams.

What test to run and where to insert them-at what readiness? At what interface? In what environment? Against what data? When? These are all questions that will become the foundation of your Continuous Delivery Testing strategy.  There are 3 elements that form your strategy:

  • Environments
  • Regression Automation Suites
  • Testing in Production


In Agile or Scrum, Dev or Test Teams “progressed” through environments; in Continuous Testing, it’s important to consider what and why we test on different environments:

In DevOps/CD, you might be running your Automation on many or all these environments concurrently. In DevOps/CD, you might be running your Automation on many or all these environments concurrently to save time and get more immediate feedback on quality. When developing your testing strategy, consider the following questions:

  • What tests need to run on the built environment? Why? 
  • What tests get to run in the test environment? Why? 
  • What are the target environments of Automated tests? E.g. VMs with various configurations, different OS versions, different languages, browsers, mobile devices, etc.? 
  • What Automated tests get to run on the staging server? What is its purpose? 
  • What gets run in production? What is its purpose?

Usually, the early stages in the DevOps pipeline such as build, test, and staging environments are where we mostly focus on catching coding bugs, UI, and functional issues. Later stages in the pipeline focus more on user-acceptance tests (end-to-end happy paths), configuration management mistakes, and performance testing.

Regression Automation Suites

The term regression is probably the most misunderstood in all software development. The fact is, regression means many things to different people. Rethinking regression and Automation is a big task for even the most effective teams. The goal of Continuous Delivery and DevOps is to provide immediate feedback (aka principle 4 – deliver as fast as possible).

To set the record straight, a regression test is any test you run again. Some people in development think the regression suite or full regression is running every test run against a previous build or product. That is rarely the case. Their regression suite is most likely re-running every Automated test. For some teams, their automated “full regression” suite can take days to run. Many “full regression” suites have gotten bloated and slow–which is not okay. There is a speed to Continuous Delivery and Continuous Deployment that leaves no room for “backup plans.” The new description of regression has to be “Lean and mean,” fast and furious, bug finding! Not bloated or the same old “whatever” we automated at the time.  So, for teams that are looking to implement CD, cleaning out their test suites to be Lean and mean is a priority.

Testing in Production

It has always been a strict rule: “No testing in production.” The reason here is that the last thing you wanted was to have the test team mess around on a live system or environment with customers, only to find out that the testing did something to muck up the system.

This has entirely changed in Continuous Testing. In CD, you may be asked to run a suite of tests to validate the migration to live or monitor the live system. Whatever the case, you need to make sure you are testing what needs to be tested and no more than you need to on live systems. 

Make sure the whole team knows the risks of testing in the live production environment, the risk of not testing in the live environment, as well as that tests cannot interfere with the normal user’s workflow performance or operation of the system-be transparent.

Continuous Delivery Best Practices

We’ve identified 5 best practices for implementing Continuous Testing into your Software Development Life Cycle to not only improve your testing, but also your overall software quality. They are:

  1. Know the business goals and processes
  2. Excellent collaboration between developers and testers
  3. Choose the best software for your niche
  4. Incorporate appropriate quality analysis
  5. Make it automated

To get a better understanding of each of these, as well as their implications for Continuous Delivery, check out Hans Buwalda’s article, 5 Best Practices for Continuous Testing.

Getting Started with Continuous Delivery

Continuous Delivery requires both high test coverage levels and an assurance that their practices are in good shape. It is important that businesses do not underestimate the difficulties of Automation. According to the testing essentials survey, released by LogiGear Magazine, almost half of respondents answered that the biggest phase or aspect of testing that is not understood by management is that “Test Automation is not easy.” Automation, and to a larger extent DevOps, can’t be successful if there isn’t serious time and investment from the start.

According to the Test Automation survey released by LogiGear Magazine, only 19% of respondents who did a Test Automation implementation were able to do it successfully the first time. The Test Automation survey also revealed another shocking insight: When asked what would prevent them from Automating more, the top five results were:

  1. “Not enough technical skill to build more successful Automation.” (34% of respondents)
  2. “Investment in Automation programs (training, staff, tool maintenance) is too high.” (32% of respondents)
  3. “Management does not understand what it takes to successfully automate.” ( 31% of respondents)
  4. “Tool use cost is high.” (31% of respondents)
  5. “Code or UI is too dynamic to automate more.” (31% of respondents)

When looking at these problems, they largely deal with either staffing or tool use. Commitment to the toolchain, as mentioned earlier, is often expensive. While there are many tools on the market, there are open source solutions like Selenium, or our own Testarchitect Enterprise and TestArchitect Gondola that can be used for Continuous Testing. The TestArchitect Automation suite integrates with most leading CI/CD tools including Microsoft Team Foundation Server, Azure DevOps Services, Azure DevOps Server, HP Quality Center, Atlassian Jira, Jenkins, and more!

Both TestArchitect Enterprise and Gondola are closed-source tools, but they are available for free! Both tools follow the Keyword-Driven Testing approach, specifically the Action-Based Testing method (ABT). ABT focuses on high-level test designs and automating keywords, not troubleshooting technical problems. 

Since keywords are either easily implemented or pre-built out-of-the-box, you can speed up your test development tremendously. On top of that, since keywords are highly reused among test cases, you save a lot of time and energy in unbreaking your flaky tests when new changes to the UI and business logic occur.

For staffing, there are several options. If you have the requisite staff in-house but are lacking the technical skill for building a successful Automation program, it may make sense to consider engaging with a testing vendor.


Hopefully, this article has proven useful in clarifying the basic concepts of Continuous Integration, Continuous Delivery, and Continuous Testing. On top of that, we hope that our audience enjoys and reaps the best benefits from the best practices in CD, tool recommendations, tips on building a good CT strategy that fits your organization.

Thuc Nguyen
Thuc Nguyen has been leading the product teams at LogiGear in delivering quality Test Automation solutions to LogiGear’s customers and services clients. Thuc has a great passion for helping organizations transform their Test Automation, Continuous Delivery and DevOps practices, as well as empowering testers of all technical levels to thrive in complex enterprise environments.
Thuc Nguyen on Linkedin

The Related Post

How to ensure a successful test-driven environment In order to ensure a higher quality product is released in the end, many teams have turned to test-driven development. Under this scenario, quality assurance metrics professionals first create various QA tests, and then software engineers code based on these tests, typically while using a robust enterprise test management ...
Making the leap to CT is easier than you think— follow this guide to transform your testing process No pain, no gain! Achieving Continuous Testing shouldn’t take a “Hans and Franz” attitude. It should be painless, more like a natural progression from implementing certain practices over time.
…On what you need to know before making the transition to EaaS 1. What are the main differences between cloud-based environments and cloud infrastructure? An environment is a collection of infrastructure elements working in conjunction to enable an application stack to work. For example, a simple 3-tier application, with a web front-end component, a business logic ...
Do you want to speed up your automated tests by a factor of 10 and deploy your application continuously? In this article we share how the JIRA development team at Atlassian has accomplished this using Stages in Bamboo. Stages have allowed the JIRA Development team to take a week’s worth of testing and condense it ...
Cloud computing has been the buzzword in the world of Information Technology for quite some time and it is likely to retain that status in the coming years. Cloud computing has been helping business enterprises deliver services faster and cheaper compared to all other existing delivery models. Small and medium business enterprises have changed their ...
LogiGear Magazine June Issue 2020: Transform Your SDLC With Continuous Testing
A test team’s job is to report test results, not set or guarantee that you will meet the SLAs. In the rush to cloud services, with everything-as-a-service, you will hear people talking about SLAs. What is this about and what does it have to do with testing? A Service Level Agreement, or SLA, is a ...
It is a fundamental role for testing teams to align their test design, test automation, and test case development with DevOps–not only to verify that code changes work but that the changes do not break the product. A key differentiator of DevOps is testing maturity. An organization can automate their integration, testing, delivery, and monitor, ...
It’s no secret that the cloud is growing at an exponential rate. By 2016, two-thirds of the world’s server workloads will exist in the cloud. But according to Cisco’s 2012 Cloud Index, less than half of server workloads currently run in the cloud. Closing the gap between current capabilities and future requirements is a mission-critical ...
In this article, I share some of my experiences and observations on training teams, mainly in corporate settings. The operative word here is “team”, not “individual”. When training teams or groups in an organization, many of the considerations and benefits are different than those for the individual. We’ll examine those differences and I will share successful solutions. ...
This post is part of the Pride & Paradev series With continuous deployment, it is common to release new software into production multiple times a day. A regression test suite, no matter how well designed, may still take over 10 minutes to run, which can lead to bottlenecks in releasing changes to production. So, do ...
By Jez Humble and David Farley Continuous Delivery from Jez Humble and David Farley is an important contribution to the field of software development. It takes continuous integration to the logical conclusion and covers how to set up a continuous integration system, delving into everything from check-in to delivery to production. It doesn’t state you ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news