DevOps has been described as Agile on Steroids; DevOps has also been described as Agile for Operations/IT. I like both of those descriptions. Many organizations want Development, Test, and Operations teams to move to DevOps now.
DevOps is a big topic, but DevOps is not the focus of this article. We will not be talking about, for example, containers, or Docker, or Puppet, or Infrastructure as code. In this article, we are going to focus on Continuous Testing. I will focus on what test teams are responsible for: The practices and concerns for testers to have their work become an asset to the team and not a drag in this exciting new world of DevOps.
This is part 2 of a 2-part series on Continuous Testing. Part 1 was focused on the rationale, business goals, culture change, and overview with some specific testing tasks. Part 2 is a deeper dive into the specific ideas, practices, and tasks of test teams that need to be updated or changed to be successful in this next phase of product development. This focuses on test strategy and Automation goals.
We gave a quick overview of the Continuous Testing Process in Part 1:
- Start with an automated smoke test. Move these into CI build process tool.
- Build bigger regression suites. Move these into CI build process tool.
- Grow in levels of awesomeness of CI; Run smoke and/or regression on multiple VMs.
- Easy and effective reporting back to the organization.
- Use containers and/or virtualization for data and full production-like environments.
- Distribute automated tests into different suites with varying goals in different environments. Use VMs for various environments to grow Automation, coverage, speed, monitoring, and feedback to the team.
As we get into this deeper discussion on Continuous Testing and Test Automation strategy, there is an additional idea to remember: DevOps is about delivering value to the customer when the business wants and needs it, not when Dev or Test or Ops gets to it.
These topics are very closely related.
- Continuous Testing Strategy
- Rethink Your Automation
- Change Your Understanding of Regression
- Embracing Lean and Mean Automated Regression Suites
Now, let’s go into more detail.
Continuous Testing Strategy
The first step towards an effective Continuous Testing strategy is understanding what exactly Continuous Testing (CT) is. By definition, Continuous Testing is a Software Testing practice that typically involves a process of testing early, testing often, testing everywhere, and automating your testing. It emphasizes on evaluating quality at every step of the Continuous Integration and Delivery process (CI/CD).
Continuous Testing is not merely testing continuously–it’s much more intelligent than that! A very common mistake is to run and rerun the same “full regression suite” on different environments and think that is enough to be “doing Continuous Testing.”
Instead, a key part of developing your Continuous Testing strategy is thinking about testing in different environments, including which environments you’ll need to test on and why you’ll test on that environment–you need to think of what and why we test on different environments. In Agile/Scrum, Dev and Test teams “progressed” through these environments. Now, in DevOps/Continuous Testing, you might be running your Automation on many or all of these environments concurrently. For example:
- What tests need to be run on the build environment?
- What tests need to be run on the test environment?
- What automated tests get run against, for example, VMs with various configurations, languages, browsers, mobile devices, etc.?
- What gets run in production?
We also must examine other questions, like, do we understand the results? Are they what we expected or the results we were looking for? Most of all, do they satisfy the client’s needs, expectations, and project requirements?
Many times, senior management judges “Software Testing success” solely based on one metric: cost. However, there are so many other relevant metrics to take into account when analyzing the success of a testing project! Management should stop seeing simply the cost of Software Testing, and instead see the value that Software Testing brings to the software development lifecycle (SDLC). When measuring the success of a testing initiative, here are some other metrics to take into account:
- The breadth of test coverage
- Test reusability
- Value of the data reported by Software Testing
- The quality of the software under test
To learn more about why these specific metrics are beneficial, check out our post on the LogiGear Blog, How to Measure Testing Success.
Rethink Test Automation
Continuous Testing relies on… Continuous Testing! Thus, everything needs to be automated–or better yet, everything that can be automated needs to be. This concept is the single key to successful Continuous Testing.
What tests need to be automated, what tests do not, how often they should be run and on which environments, how fast they are to run, the possibility that the test will “fail,” the size of tests and their maintenance… all of these factors should impact what and how things will get automated.
Many people talk about Automation in DevOps. Often, though, they do not mean Test Automation. They may be referring to Task Automation by the Ops team. Automating building environments, provisioning databases, populating user lists… These common tasks for many IT/Ops teams can now be scheduled, automated, and run significantly faster due to the flood of new tools focused on Ops and the “Infrastructure as Code” movement.
DevOps demands an entire re-thinking of your Automation program–both Test and Task Automation. For example, it is common to shrink your “full regression” to run faster and in more environments. Shrinking automation suites is an uncommon goal for many test teams, but one that finds itself incredibly useful and necessary in a CT implementation.
Change Your Understanding of Regression
As I stated above, the idea of CT being the rerunning of your full regression suite repeatedly is a common misconception. CT does involve regression; however, in order to be effective and aligned with the goals of both CT and Automation, we’re going to need to rethink and refactor our regression suite(s) from the traditional ideas of regression.
Regression testing is often intertwined with the idea of “defect priority,” which ranges from priority 1 (P1), where the defect needs to be fixed within 24 hours, to priority 4 (P4), where there is definitely a defect, but it does not need to be fixed to match the “exit” criteria. Regression is often only running P1 or P2 tests, where lower-priority tests are of less importance, not automated, and not re-run. But, regression may only be the user story validation tests that were documented and automated along the way based on how much time the Automation Engineers had to automate whatever tests they did rather than focus on long-lasting, confidence-building, bug-finding regression suites. The fact is, regression means different things to different people. This is particularly true where “automate everything” is a common mantra in Continuous Testing. There is a speed to Continuous Delivery and Continuous Deployment that leaves no room for “backup plans.”
Regression testing, by definition, “is the re-running of functional and non-functional tests to ensure that previously developed and tested software still performs after a change. Basically, it can apply to any test you have to run more than once. Developers usually test, test, and re-test as they are building the solution in a sandbox or against a “pilot”’ version of the solution; this ensures that their code is working as expected, the results are what is expected as per the requirements, the output looks reasonable, and the performance is within whatever performance limits have been set for the solution.
In CT, the need for us to be testing more often and in more environments is pretty obvious. But, the tests that we are running have to change. Understanding the purpose of each environment and the monitoring or information needed from that environment is key to selecting which tests get run. Rerunning the same tests over and over on the same system with the same data against the same build does not help anyone. We know we have a smoke test suite–a build validation suite that is run in the CI process on the Dev or Test environment. We all have a “full regression” suite that we normally run against the Test environment. Here is where we need to begin reformulating.
Embracing Lean and Mean Automated Regression Suites
For some teams, their automated “full regression” suite can take days to run. Many “full regression” suites have gotten bloated and slow. This is not okay. The new description of regression, which borrows largely from Lean Software Development’s principle of Cut Waste, is that regression now has to be “Lean and mean,” fast and furious–not bloated, and not the same old “whatever we automated last time.”
Continuous Testing suites need to be economical and efficient. Teams will need to redesign, optimize, and even cut the size of their automated suites for Continuous Testing rather than keep them bloated and slow.
Why? For a few reasons:
- Tests get old and obsolete
- Tests need maintenance
- Tests that often fail or break easily need to be rethought and redesigned to find bugs, or break faster and be less fragile.
- Receive more Immediate feedback.
- An efficient set of tests will be run by other people on the team in Continuous Testing.
- Get the biggest coverage with the smallest number of tests. That is, get the biggest bang for your buck.
You will need various suites of tests. For example, you may have a few “regression” suites:
- A fast, smoke regression,
- A big, full regression,
- A smaller, “happy path”/main functionality, bug-finding regression suite, a main or core “Lean and mean” regression suite, or a smaller end-to-end transaction/integration style regression suite for the full integration environments.
Each will be run at various points in the DevOps process in various environments. It’s important to remember, all this testing is in addition to the new feature development testing that happens inside your sprint..Will you run any of these in production? I hope not. The goals of those suites are not what you want in production! An automated suite of tests you run in production must be extra careful to maintain the integrity of the production environment and users while monitoring system behavior and availability. So, you may want to add on to the above list of “regression suites” a smaller set of whatever you choose to run against production for monitoring.
Testing in Production
For all the years I have been in testing, there has always been a strict rule: No testing in production. The last thing you want is to be messing around on a live system/environment with customers and then accidentally do something to muck up the system. I fully support this notion and strictly adhere to duplicating the data and solution on a sandbox or pilot server so we can play with whatever needs to be played with. It may result in a need to fix or re-work something in production at some point if necessary, but right is right and the customer and end-users need a solution that works and that provides the data output and performance expected-sometimes no matter the cost (although change orders–if approved–can certainly make this less painful).
Summary
Developing a test strategy for Continuous Testing is a full reexamination of what gets tested where. This with the Lean Software development (LSD) idea of Quality at Every Step is the foundation of a Continuous Testing strategy. It involves reexamining what you have been calling “regression” and developing multiple Lean and mean suites for immediate feedback.
It also means rethinking what and how you automate. Old style automation, even small isolated automated tests, need to be reexamined and replaced by efficient, low-level tests as well as longer, customer-focused transaction workflows. Test early, test often is the best solution, and Automated Testing–to whatever degree possible–can work well in the delivery team’s favor to remove the human error in the process and greatly speed up the testing process and timeline to production readiness we are all focusing on.
