DevOps for Test Teams
By Michael Hackett
Now that Dev Teams have had a little time to settle into the Agile, the new wave of process optimization has arrived. DevOps. DevOps has been described as Agile on Steroids. DevOps has also been described as Agile for Operations/IT. I like both of those descriptions as well as some others. Many organizations want Development, Test and Operations teams to move to DevOps now.
DevOps is a big topic. But DevOps is not the focus of this article. We will not be talking about, for example, containers, or Docker, or Puppet or Infrastructure as code. In this article, we are going to focus on Continuous Testing. I will focus on what Test teams are responsible for–the practices and concerns for testers to have their work become an asset to the team and not a drag in this exciting new world of DevOps.
This is a 2 part series on Continuous Testing. Part 1 is on the rationale, business goals and overview with some specific testing tasks. Part 2 is a deeper dive into the specific ideas and tasks of Test teams that need to be updated or changed to be successful in this next phase of product development.
There are many problems DevOps is trying to solve. The overriding idea is to:
1) Shift Operations or IT tasks left, earlier in the process.
2) To leverage a bunch of newer tools and technologies to automate operations tasks, like provisioning and migrating code, leveraging these tools.
3) Have everyone on the product team communicate and collaborate much more and earlier.
4) To have the product be ready to be deployed—in a consistent state of readiness so that the business can decide when new functionality goes to customers. And ultimately, not be held back by Dev or Ops being unstable and not prepared to go with whatever is ready.
Continuous Testing gets us far along this path and is the part of DevOps most concerning to traditional Test teams. That is, Test teams that would execute everything from black box, to regression, to UI, to workflow scenario tests. Who generally do not do security, performance or unit tests and have therefore not executed tests in production.
When I think of Continuous Testing I think of the Lean principle, quality at every step. When developers commit new code, test it! When the product gets integrated, test it! When the product gets moved to any new environment, like the test environment, staging environment or production, test it!
There are Continuous Testing activities.
Over the last few years, with Agile and Continuous Integration, we have already shifted left and partially right. Now, we have to learn how to shift testing fully right. To do this, and test on all the various environments and stages that need to be tested and validated, and for rapid delivery and/or deployment, the testing, of course, needs to be automated.
Another way to look at this is as a system, there are three parts:
- The build delivered by Development.
- The deployment on infrastructure handled by OPS/IT.
- The customers who use the apps/system.
Test teams must have strong knowledge of these three parts, know the goals of testing at each part, be able to design great tests for each part, and have the ability to automate these tests. This is harder and more comprehensive than most organizations think.
Many Test teams still struggle with Agile, fast test automation and automation maintenance. These problems will not help the organization in its mission; it will only drag it down. Tests can be designed and built for successful Continuous Integration first, and expand that to Continuous Testing by knowing the right things to do, staying focused on a few key points and most importantly, automating as smartly as possible.
Let’s get started with the work.
The first task. Know what you are talking about with DevOps.
Go read about DevOps. Familiarize yourself with the terms currently used in discussions about DevOps, you can also reference our Glossary in the following pages.
For example, Continuous Delivery is not Continuous Deployment. Find out the details, the tools Ops/IT will be using and why they will use them. Learn about provisioning, containers, virtualization, and automatic deployment. We need to be familiar with the whole DevOps paradigm to be useful to the team and also to design the right tests for each situation, even though they may not be part of our job. For example—understanding containers may impact our test design for integration testing—but the how, when and where to use containers is not a Test teams task.
For everyone, DevOps is an automatic leveraging technology, in which everyone’s tasks are automated. From builds, to provisioning, to deployment, and everything in between—it’s all automatic. The goal is for Test teams to automate as effectively, thoroughly, and economically as possible. This will not be easy, nor is it an instant change.
DevOps for testers is focused on Continuous Testing.
The goal of Continuous Testing is that these test tasks provide immediate, useful feedback to the whole team on the stability and consistency of the product, ultimately giving the team confidence before delivery and deployment.
Test teams must also move seamlessly through the testing cycles—this is a constant rolling process. For most teams who do DevOps, “phases” is not used to describe where the product is or its readiness. The product can be in Dev, in Test, tests running on staging server, and tests running in production at the same time. If the business decides to deliver functionality to customers, it can move delivered code to deployment based on feedback from the tests, instead of the phase the product is in. Essentially, making the product is in a constant state of test. This will be the primary factor in building consistency from coding to deployment.
“Testing” is now about all the test efforts.
Of course, first, you need awesome testing. You will need more of it, it will need to be Agile and it will need to be automated. Automated more than ever before. Testing includes re-running unit tests, running functional, workflow, validation, and bug-finding tests. These along with security tests, load and performance testing…there is no separation in terms of running these tests for consistency, monitoring and reporting. When people describe DevOps as a “shift left,” one reference is to running performance tests early and not waiting until the end of development. This is an important example because it illustrates the many pieces of DevOps. The production environment can be virtualized and spun up at any moment with a VM or in the cloud and have full performance tests run early in development. In the very recent past, good Developers would run performance tests early but may have had to mock up a limited system and focus on isolated performance tests.
Different automated suites of combinations of these tests will be run on various environments throughout the process. Depending on your organization, some tests “belong to” the QA or Test team. You may be expected to take on more test and monitoring responsibilities.
Testing in production is normal and expected. This may be problematic—be intelligent and careful.
Now testing happens everywhere, and is done by everyone. To be successful in DevOps, aside from test automation, the test leadership should have assigned Developers, Test Operations and QA leadership as part of the same team. Testing is now whole, no longer just a part.
Your team has to be great at Agile. Scrumbutts and AgileFalls will fail in DevOps.
As said above, some people describe DevOps as Agile on Steroids. It you aren’t great at Agile already, Continuous Testing will fail, and DevOps will fail. If there are problem areas in your Agile/Scrum process—fix them before moving any further.
Also, if you are having a difficult time presently keeping up with automation in your sprints, any attempts at Continuous Testing will surely fail. Your testing, test automation and reporting must already be smooth, low maintenance, and effective.
The test automation of new functions has to happen within the sprint; first manual testing in sprint and then by the end of the sprint.
Don’t even try Continuous Testing until you have great, meaningful, effective, fast Continuous Integration.
In every team I have worked with, CI leads to Continuous Testing. CI is the foundation of Continuous Testing.
Having a great Agile practice means the organization understands everyone does Quality Assurance, not testing. Quality Assurance means quality user stories, quality unit tests, and quality environments. Quality at every step.
You need a better, more comprehensive test strategy.
Get your test game on! Great testing skills are still your most important job skill.
Where old style “QA” testing tested at the end, Continuous Testing takes Test teams and entire organizations to a whole new level.
Continuous Testing, in theory, frees you from focusing on big bang deployments. You have automated suites running constantly to monitor system health as more work items are delivered. You will need a strategy for testing in the sprint, a strategy for what to test in CI, and a strategy for what to test in the various environments.
Test teams need to stay focused on delivering Done, potentially shippable User Stories at the end of each sprint or iteration as well as growing and optimizing the various regression suites.
You still need to adequately test new functionality. Don’t focus too much on small “acceptance” tests. You need workflows and integration tests, user scenarios and task based tests. Low-level, small tests may miss workflow/integration/end-to-end problems.
You need speed but don’t forget exploratory, data driven, and soap opera tests. Too many teams get overwhelmed by more automation and focus only on test automation at the expense of a full bug finding, validation, design collaboration, and customer experience test strategy.
You may have a separate team building the test automation framework and another team doing automation to pick up the pace. Your strategy will need more low-level tests. Integration tests, especially in the new “container” world where containers/components can be easily plugged in, are high priority.
Another trend in software development over the last few years is Service-Oriented Architecture (SOA). Today, SOA has gone into overdrive, for many reasons not important here, but with the use of containers and microservices as a goal of many organizations. Whether your team uses or refers to, APIs, SOA, web services, or microservices; or is using a container tool like Docker, more testing is happening now on more lower-level items, and is more technical. Testing isolated services is great for finding bugs or breaks fast. Then, separately, integration tests need to run for correct integration and consistency.
Testing should know a lot about grey-box testing. This is about grey-box, intelligent test automation with knowledge to design better tests and to do quicker failure analysis.
An analogy of testing in DevOps is similar to an assembly line production, like a car manufacturing line, for example. You need to check and test everywhere—not only at the end. A key difference in a car assembly line, though, is that the parts are fixed. With software in DevOps, the parts are changeable and when the parts change, they will need automated testing to keep up.
Continuous Testing in DevOps requires culture as well as process change. Part 1 of this article focused on culture and understanding. Part 2 will focus more on process and tasks.
A quick, Continuous Testing task overview is:
- Start with an automated smoke test. Move these into CI build process tool.
- Build bigger regression suites. Move these into CI build process tool.
- Grow in levels of awesomeness of CI; Run smoke and/or regression on multiple VMs.
- Easy and effective reporting back to the organization.
- Use containers and/or virtualization for data and full production like environments.
- Distribute automated tests into different suites with varying goals on different environments. Use VMs for various environments to grow automation, coverage, speed, monitoring and feedback to the team.
Continuous Testing grows from this.
Part 2 of this topic will define and describe the lower level practices and automation goals, environment and data problems to solve, virtualization, with more details on testing goals.
