Disclaimer: This article was originally published in the LogiGear Magazine, but has been updated for the June 2020 issue.
Leading an organization into Continuous Delivery is a daunting task that usually takes software development teams months if not years. This guide covers the main points that organizations need to consider to increase their chance of success when adopting Continuous Delivery. Topics to be covered include:
- Continuous Integration vs. Continuous Delivery
- Continuous Delivery Pipeline
- Continuous Delivery Tools
- Continuous Delivery Testing
- Continuous Delivery Test Strategy
- Continuous Delivery Best Practices
- How to Get Started with Continuous Delivery
The fundamental underpinnings of CD are dependent on 2 parts: culture change and commitments to the toolchain. Typically, there is more emphasis on commitment to the toolchain; this is because CD requires an increase in Automation–both Task Automation and Test Automation. Each section contains everything businesses need to consider when it comes to building and implementing a successful Continuous Testing program, from the toolchain to best practices, developing a strategy, and discussing options on how to get started.
Continuous Integration vs. Continuous Delivery
First, let’s establish the definitions of Continuous Delivery and Continuous Integration and establish the differences. Continuous Integration is a software engineering practice in which the changes made by developers to working copies of code are added to the mainline code base on a frequent basis, and immediately tested. The goal is to provide rapid feedback so that if a defect is introduced into the mainline, it can be identified quickly and corrected as soon as possible.
Continuous Integration software tools are often used to automate testing and build a document trail. Because CI detects deficiencies early on in development, defects are typically smaller, less complex, and easier to resolve. In the end, well-implemented CI reduces the cost of software development and reduces the time to market.
Contrasted, Continuous Delivery is a “Software Engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software faster and more frequently. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for Continuous Delivery.”
The main difference is that CI focuses on Dev and Test teams, whereas CD focuses on Ops teams. Where CI ends, CD picks up: promoting builds, versions, and deployments up closer to and finally, the live/production environments.
Continuous Delivery Pipeline
Continuous Delivery has been related to a production pipeline. The main idea is that the “Continuous Delivery pipeline” is the image and description of product flow through the development process and toolchain. This pipeline flow is a variation of a manufacturing production line.
It became very clear in Agile that quality is everyone’s job. Likewise, everyone owns a piece of quality. In CD, everyone owns deployment and everyone shares a piece of deployment. Many things must change about the product to get these ideas implemented: architectural changes, tool changes, learning new skills to facilitate culture changes and rethinking the idea of release.
As with all production lines, there will be bottlenecks. The goal with pipelines is a constant flow of high quality and finished/deployment-ready work. Keep the flow going. Keep the pipeline moving. Get lots of immediate feedback to the team.
The Continuous Delivery pipeline is a huge topic within itself, and it could be written about extensively. In fact, Michael Hackett already has! Check out Michael’s article, Mega Trends in Testing: Continuous Delivery, Production Line, and the Deployment Pipeline, to get a full, in-depth understanding of the CD pipeline.
Continuous Delivery Tools
The financial commitments to the toolchain often require a large upfront investment for organizations, this can be hard due to the necessity of increasing task Automation. Tools will be needed for the following areas:
- Cloud/EaaS, Iaas, Paas
- Configuration Management (e.g. Chef and Puppet)
- Containers (e.g. Docker and Packer)
- CI/Version Control (e.g. Jenkins, Bamboo, TeamCity, and Buildbot)
- Deployment (e.g. XebiaLabs Deploy, Go Thoughtworks, IBM UrbanCode Deploy, and Code Stream)
- Monitoring (e.g. New Relic and App Dynamics)
- Source Control (e.g. Git SVN)
- Test Automation (e.g. Selenium, TestArchitect, Cucumber, and jBehave)
As you can see, for each different type of tool, there are many different options. At the basic level, they all perform the same task, but which one is right for you depends on your organization’s needs and processes, as well as your Automation Tool of choice. Make sure that you do proper research into prospective tools before beginning your CI/CD implementation.
Continuous Delivery Testing / Testing in Continuous Delivery
The figure below illustrates all the places where Test Automation appears in the Continuous Delivery Pipeline.
In Continuous Delivery, Testing appears at every stage of the pipeline–it is no longer isolated. It is important to note that test teams, including Test Automation, cannot become the bottleneck in the Continuous Delivery pipeline. The main reason is that if Software Testing or Test Automation does become a bottleneck, it will be ignored and quality will certainly deteriorate.
For the product pipeline to keep moving with immediate feedback, you don’t have to take one Automated UI regression suite and have it run again, and again, and again. That doesn’t ensure quality or confidence. There are certain times, particularly with newly committed code, that unit tests are the most efficient for immediate feedback. There are other times where API or service level testing are the best strategies, and still, other places where the tests best inserted into the pipeline are UI-driven tests. Testing in Continuous Delivery comes with many perks including higher efficiency and higher ROI, but it’s not an easy implementation; there are many aspects that can’t be covered in one magazine article, and that’s why we wrote our eBook Testing in Continuous Delivery. It features in-depth industry knowledge that leader’s need to know in order to successfully lead their team into Continuous Delivery.
Continuous Delivery Test Strategy
What is commonly referred to as a “Continuous Delivery Test Strategy” is in actuality Continuous Testing? Continuous Testing (CT) is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate, and CT has its own Automation strategy. It is important to remember that as you build test suites for the CD pipeline, Test Automation does not show a lack of bugs. Test Automation shows consistency with the last iteration; it shows stability and predictability. Not finding bugs does not indicate that there are no bugs-it indicates that the bugs that haven’t been found yet are still hidden.
The test strategy for Continuous Testing is a full reexamination of what gets tested where. This with the Lean Software development (LSD) idea of “quality at every step” is the foundation of a CD test strategy. It involves reexamining, what you have been calling, “regression” and developing multiple lean and mean suites for immediate feedback.
It also means rethinking what and how you automate. Old style Automation-even small isolated automated tests need to be reexamined and replaced by efficient, low-level tests as well as longer customer-focused transaction workflows. Rethinking regression and Automation is a big task for even the most effective teams.
What test to run and where to insert them-at what readiness? At what interface? In what environment? Against what data? When? These are all questions that will become the foundation of your Continuous Delivery Testing strategy. There are 3 elements that form your strategy:
- Environments
- Regression Automation Suites
- Testing in Production
Environments
In Agile or Scrum, Dev or Test Teams “progressed” through environments; in Continuous Testing, it’s important to consider what and why we test on different environments:
In DevOps/CD, you might be running your Automation on many or all these environments concurrently. In DevOps/CD, you might be running your Automation on many or all these environments concurrently to save time and get more immediate feedback on quality. When developing your testing strategy, consider the following questions:
- What tests need to run on the built environment? Why?
- What tests get to run in the test environment? Why?
- What are the target environments of Automated tests? E.g. VMs with various configurations, different OS versions, different languages, browsers, mobile devices, etc.?
- What Automated tests get to run on the staging server? What is its purpose?
- What gets run in production? What is its purpose?
Usually, the early stages in the DevOps pipeline such as build, test, and staging environments are where we mostly focus on catching coding bugs, UI, and functional issues. Later stages in the pipeline focus more on user-acceptance tests (end-to-end happy paths), configuration management mistakes, and performance testing.
Regression Automation Suites
The term regression is probably the most misunderstood in all software development. The fact is, regression means many things to different people. Rethinking regression and Automation is a big task for even the most effective teams. The goal of Continuous Delivery and DevOps is to provide immediate feedback (aka principle 4 – deliver as fast as possible).
To set the record straight, a regression test is any test you run again. Some people in development think the regression suite or full regression is running every test run against a previous build or product. That is rarely the case. Their regression suite is most likely re-running every Automated test. For some teams, their automated “full regression” suite can take days to run. Many “full regression” suites have gotten bloated and slow–which is not okay. There is a speed to Continuous Delivery and Continuous Deployment that leaves no room for “backup plans.” The new description of regression has to be “Lean and mean,” fast and furious, bug finding! Not bloated or the same old “whatever” we automated at the time. So, for teams that are looking to implement CD, cleaning out their test suites to be Lean and mean is a priority.
Testing in Production
It has always been a strict rule: “No testing in production.” The reason here is that the last thing you wanted was to have the test team mess around on a live system or environment with customers, only to find out that the testing did something to muck up the system.
This has entirely changed in Continuous Testing. In CD, you may be asked to run a suite of tests to validate the migration to live or monitor the live system. Whatever the case, you need to make sure you are testing what needs to be tested and no more than you need to on live systems.
Make sure the whole team knows the risks of testing in the live production environment, the risk of not testing in the live environment, as well as that tests cannot interfere with the normal user’s workflow performance or operation of the system-be transparent.
Continuous Delivery Best Practices
We’ve identified 5 best practices for implementing Continuous Testing into your Software Development Life Cycle to not only improve your testing, but also your overall software quality. They are:
- Know the business goals and processes
- Excellent collaboration between developers and testers
- Choose the best software for your niche
- Incorporate appropriate quality analysis
- Make it automated
To get a better understanding of each of these, as well as their implications for Continuous Delivery, check out Hans Buwalda’s article, 5 Best Practices for Continuous Testing.
Getting Started with Continuous Delivery
Continuous Delivery requires both high test coverage levels and an assurance that their practices are in good shape. It is important that businesses do not underestimate the difficulties of Automation. According to the testing essentials survey, released by LogiGear Magazine, almost half of respondents answered that the biggest phase or aspect of testing that is not understood by management is that “Test Automation is not easy.” Automation, and to a larger extent DevOps, can’t be successful if there isn’t serious time and investment from the start.
According to the Test Automation survey released by LogiGear Magazine, only 19% of respondents who did a Test Automation implementation were able to do it successfully the first time. The Test Automation survey also revealed another shocking insight: When asked what would prevent them from Automating more, the top five results were:
- “Not enough technical skill to build more successful Automation.” (34% of respondents)
- “Investment in Automation programs (training, staff, tool maintenance) is too high.” (32% of respondents)
- “Management does not understand what it takes to successfully automate.” ( 31% of respondents)
- “Tool use cost is high.” (31% of respondents)
- “Code or UI is too dynamic to automate more.” (31% of respondents)
When looking at these problems, they largely deal with either staffing or tool use. Commitment to the toolchain, as mentioned earlier, is often expensive. While there are many tools on the market, there are open source solutions like Selenium, or our own Testarchitect Enterprise and TestArchitect Gondola that can be used for Continuous Testing. The TestArchitect Automation suite integrates with most leading CI/CD tools including Microsoft Team Foundation Server, Azure DevOps Services, Azure DevOps Server, HP Quality Center, Atlassian Jira, Jenkins, and more!
Both TestArchitect Enterprise and Gondola are closed-source tools, but they are available for free! Both tools follow the Keyword-Driven Testing approach, specifically the Action-Based Testing method (ABT). ABT focuses on high-level test designs and automating keywords, not troubleshooting technical problems.
Since keywords are either easily implemented or pre-built out-of-the-box, you can speed up your test development tremendously. On top of that, since keywords are highly reused among test cases, you save a lot of time and energy in unbreaking your flaky tests when new changes to the UI and business logic occur.
For staffing, there are several options. If you have the requisite staff in-house but are lacking the technical skill for building a successful Automation program, it may make sense to consider engaging with a testing vendor.
Conclusion
Hopefully, this article has proven useful in clarifying the basic concepts of Continuous Integration, Continuous Delivery, and Continuous Testing. On top of that, we hope that our audience enjoys and reaps the best benefits from the best practices in CD, tool recommendations, tips on building a good CT strategy that fits your organization.
