Continuous Testing, Part 2: Strategy and Automation Goals for Test Teams

DevOps has been described as Agile on Steroids; DevOps has also been described as Agile for Operations/IT. I like both of those descriptions. Many organizations want Development, Test, and Operations teams to move to DevOps now.

DevOps is a big topic, but DevOps is not the focus of this article. We will not be talking about, for example, containers, or Docker, or Puppet, or Infrastructure as code. In this article, we are going to focus on Continuous Testing. I will focus on what test teams are responsible for:  The practices and concerns for testers to have their work become an asset to the team and not a drag in this exciting new world of DevOps.

This is part 2 of a 2-part series on Continuous Testing. Part 1 was focused on the rationale, business goals, culture change, and overview with some specific testing tasks. Part 2 is a deeper dive into the specific ideas, practices, and tasks of test teams that need to be updated or changed to be successful in this next phase of product development. This focuses on test strategy and Automation goals.

We gave a quick overview of the Continuous Testing Process in Part 1:

  1. Start with an automated smoke test. Move these into CI build process tool.
  2. Build bigger regression suites. Move these into CI build process tool.
  3. Grow in levels of awesomeness of CI; Run smoke and/or regression on multiple VMs.
  4. Easy and effective reporting back to the organization.
  5. Use containers and/or virtualization for data and full production-like environments.
  6. Distribute automated tests into different suites with varying goals in different environments. Use VMs for various environments to grow Automation, coverage, speed, monitoring, and feedback to the team.

As we get into this deeper discussion on Continuous Testing and Test Automation strategy, there is an additional idea to remember: DevOps is about delivering value to the customer when the business wants and needs it, not when Dev or Test or Ops gets to it.

These topics are very closely related.

  • Continuous Testing Strategy
  • Rethink Your Automation
  • Change Your Understanding of Regression
  • Embracing Lean and Mean Automated Regression Suites

Now, let’s go into more detail.

Continuous Testing Strategy

The first step towards an effective Continuous Testing strategy is understanding what exactly Continuous Testing (CT) is. By definition, Continuous Testing is a Software Testing practice that typically involves a process of testing early, testing often, testing everywhere, and automating your testing. It emphasizes on evaluating quality at every step of the Continuous Integration and Delivery process (CI/CD).

Continuous Testing is not merely testing continuously–it’s much more intelligent than that! A very common mistake is to run and rerun the same “full regression suite” on different environments and think that is enough to be “doing Continuous Testing.” 

Instead, a key part of developing your Continuous Testing strategy is thinking about testing in different environments, including which environments you’ll need to test on and why you’ll test on that environment–you need to think of what and why we test on different environments. In Agile/Scrum, Dev and Test teams “progressed” through these environments. Now, in DevOps/Continuous Testing, you might be running your Automation on many or all of these environments concurrently. For example:

  • What tests need to be run on the build environment? 
  • What tests need to be run on the test environment? 
  • What automated tests get run against, for example, VMs with various configurations, languages, browsers, mobile devices, etc.? 
  • What gets run in production? 

We also must examine other questions, like, do we understand the results? Are they what we expected or the results we were looking for? Most of all, do they satisfy the client’s needs, expectations, and project requirements?

Many times, senior management judges “Software Testing success” solely based on one metric: cost. However, there are so many other relevant metrics to take into account when analyzing the success of a testing project! Management should stop seeing simply the cost of Software Testing, and instead see the value that Software Testing brings to the software development lifecycle (SDLC). When measuring the success of a testing initiative, here are some other metrics to take into account:

  • The breadth of test coverage
  • Test reusability
  • Value of the data reported by Software Testing
  • The quality of the software under test

To learn more about why these specific metrics are beneficial, check out our post on the LogiGear Blog, How to Measure Testing Success.

Rethink Test Automation

Continuous Testing relies on… Continuous Testing! Thus, everything needs to be automated–or better yet, everything that can be automated needs to be. This concept is the single key to successful Continuous Testing.

What tests need to be automated, what tests do not, how often they should be run and on which environments, how fast they are to run, the possibility that the test will “fail,” the size of tests and their maintenance… all of these factors should impact what and how things will get automated. 

Many people talk about Automation in DevOps. Often, though, they do not mean Test Automation. They may be referring to Task Automation by the Ops team. Automating building environments, provisioning databases, populating user lists… These common tasks for many IT/Ops teams can now be scheduled, automated, and run significantly faster due to the flood of new tools focused on Ops and the “Infrastructure as Code” movement.

DevOps demands an entire re-thinking of your Automation program–both Test and Task Automation. For example, it is common to shrink your “full regression” to run faster and in more environments. Shrinking automation suites is an uncommon goal for many test teams, but one that finds itself incredibly useful and necessary in a CT implementation.

Change Your Understanding of Regression

As I stated above, the idea of CT being the rerunning of your full regression suite repeatedly is a common misconception. CT does involve regression; however, in order to be effective and aligned with the goals of both CT and Automation, we’re going to need to rethink and refactor our regression suite(s) from the traditional ideas of regression.

Regression testing is often intertwined with the idea of “defect priority,” which ranges from priority 1 (P1), where the defect needs to be fixed within 24 hours, to priority 4 (P4), where there is definitely a defect, but it does not need to be fixed to match the “exit” criteria. Regression is often only running P1 or P2 tests, where lower-priority tests are of less importance, not automated, and not re-run. But, regression may only be the user story validation tests that were documented and automated along the way based on how much time the Automation Engineers had to automate whatever tests they did rather than focus on long-lasting, confidence-building, bug-finding regression suites. The fact is, regression means different things to different people. This is particularly true where “automate everything” is a common mantra in Continuous Testing. There is a speed to Continuous Delivery and Continuous Deployment that leaves no room for “backup plans.” 

Regression testing, by definition, “is the re-running of functional and non-functional tests to ensure that previously developed and tested software still performs after a change. Basically, it can apply to any test you have to run more than once. Developers usually test, test, and re-test as they are building the solution in a sandbox or against a “pilot”’ version of the solution; this ensures that their code is working as expected, the results are what is expected as per the requirements, the output looks reasonable, and the performance is within whatever performance limits have been set for the solution.

In CT, the need for us to be testing more often and in more environments is pretty obvious. But, the tests that we are running have to change. Understanding the purpose of each environment and the monitoring or information needed from that environment is key to selecting which tests get run.  Rerunning the same tests over and over on the same system with the same data against the same build does not help anyone. We know we have a smoke test suite–a build validation suite that is run in the CI process on the Dev or Test environment. We all have a “full regression” suite that we normally run against the Test environment. Here is where we need to begin reformulating.

Embracing Lean and Mean Automated Regression Suites

For some teams, their automated “full regression” suite can take days to run. Many “full regression” suites have gotten bloated and slow. This is not okay. The new description of regression, which borrows largely from Lean Software Development’s principle of Cut Waste, is that regression now has to be “Lean and mean,” fast and furious–not bloated, and not the same old “whatever we automated last time.” 

Continuous Testing suites need to be economical and efficient. Teams will need to redesign, optimize, and even cut the size of their automated suites for Continuous Testing rather than keep them bloated and slow. 

Why? For a few reasons:

  • Tests get old and obsolete
  • Tests need maintenance
  • Tests that often fail or break easily need to be rethought and redesigned to find bugs, or break faster and be less fragile.
  • Receive more Immediate feedback.
  • An efficient set of tests will be run by other people on the team in Continuous Testing.
  • Get the biggest coverage with the smallest number of tests. That is, get the biggest bang for your buck.

You will need various suites of tests. For example, you may have a few “regression” suites:

  • A fast, smoke regression, 
  • A big, full regression,
  • A smaller, “happy path”/main functionality, bug-finding regression suite, a main or core “Lean and mean” regression suite, or a smaller end-to-end transaction/integration style regression suite for the full integration environments. 

Each will be run at various points in the DevOps process in various environments. It’s important to remember, all this testing is in addition to the new feature development testing that happens inside your sprint..Will you run any of these in production? I hope not. The goals of those suites are not what you want in production! An automated suite of tests you run in production must be extra careful to maintain the integrity of the production environment and users while monitoring system behavior and availability. So, you may want to add on to the above list of “regression suites” a smaller set of whatever you choose to run against production for monitoring.

Testing in Production

For all the years I have been in testing, there has always been a strict rule: No testing in production. The last thing you want is to be messing around on a live system/environment with customers and then accidentally do something to muck up the system. I fully support this notion and strictly adhere to duplicating the data and solution on a sandbox or pilot server so we can play with whatever needs to be played with. It may result in a need to fix or re-work something in production at some point if necessary, but right is right and the customer and end-users need a solution that works and that provides the data output and performance expected-sometimes no matter the cost (although change orders–if approved–can certainly make this less painful).


Developing a test strategy for Continuous Testing is a full reexamination of what gets tested where. This with the Lean Software development (LSD) idea of Quality at Every Step is the foundation of a Continuous Testing strategy. It involves reexamining what you have been calling “regression” and developing multiple Lean and mean suites for immediate feedback.

It also means rethinking what and how you automate. Old style automation, even small isolated automated tests, need to be reexamined and replaced by efficient, low-level tests as well as longer, customer-focused transaction workflows. Test early, test often is the best solution, and Automated Testing–to whatever degree possible–can work well in the delivery team’s favor to remove the human error in the process and greatly speed up the testing process and timeline to production readiness we are all focusing on.

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

It is a fundamental role for testing teams to align their test design, test automation, and test case development with DevOps–not only to verify that code changes work but that the changes do not break the product. A key differentiator of DevOps is testing maturity. An organization can automate their integration, testing, delivery, and monitor, ...
Introduction Everything changes. It’s the only constant. The landscape of software testing is undergoing a fast and dramatic change driven by societal, business and technology changes that have placed software everywhere. Computing is ubiquitous. There is hardly anything today that doesn’t contain a CPU or information transmission capability, with software to drive it. From smart toasters ...
If you haven’t already caught a glimpse, we announced in January the Testing in DevOps and Continuous Testing videos series, available on YouTube!
A test team’s job is to report test results, not set or guarantee that you will meet the SLAs. In the rush to cloud services, with everything-as-a-service, you will hear people talking about SLAs. What is this about and what does it have to do with testing? A Service Level Agreement, or SLA, is a ...
Sauce Lab’s Perspective on Integration with Continuous Testing The term ‘DevOps’ implies that implementing an effective development and production workflow is as easy as developing the collaborative interaction between developers and IT operations. On the surface, DevOps may appear as a simple and straightforward idea but, as you will find out, there is more than meets ...
How Halliburton leveraged outsourcing to achieve their goals. Organizations are focusing on speed, both in Continuous Integration and rapid deployment as a competitive advantage. Many software development organizations can significantly shorten development cycles by implementing one or a combination of Agile practices, continuous integration & deployment methods, and feature branches. While these frameworks and techniques ...
Over the years we’ve provided an extensive number of articles, videos, and infographics that provide a wealth of knowledge about Continuous Delivery.
Fitting QA into a modern DevOps group In a traditional software engineering organization, the QA group is often seen as separate from the Development group. Developers and testers have different roles, different responsibilities, different job descriptions, and different management. They are two distinct entities. However, for folks outside the engineering team – say in Operations ...
Special considerations that should be applied to an application running in the cloud. Over the last weeks, I have found myself in several rather intense discussions about “cloud testing”: what it is, what it isn’t, and what it means for testing and QA professionals. The major source of confusion in these discussions usually revolves around ...
Cloud computing has been the buzzword in the world of Information Technology for quite some time and it is likely to retain that status in the coming years. Cloud computing has been helping business enterprises deliver services faster and cheaper compared to all other existing delivery models. Small and medium business enterprises have changed their ...
…On what you need to know before making the transition to EaaS 1. What are the main differences between cloud-based environments and cloud infrastructure? An environment is a collection of infrastructure elements working in conjunction to enable an application stack to work. For example, a simple 3-tier application, with a web front-end component, a business logic ...
Making the leap to CT is easier than you think— follow this guide to transform your testing process No pain, no gain! Achieving Continuous Testing shouldn’t take a “Hans and Franz” attitude. It should be painless, more like a natural progression from implementing certain practices over time.

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news