Continuous Testing, Part 2: Strategy and Automation Goals for Test Teams

DevOps has been described as Agile on Steroids; DevOps has also been described as Agile for Operations/IT. I like both of those descriptions. Many organizations want Development, Test, and Operations teams to move to DevOps now.

DevOps is a big topic, but DevOps is not the focus of this article. We will not be talking about, for example, containers, or Docker, or Puppet, or Infrastructure as code. In this article, we are going to focus on Continuous Testing. I will focus on what test teams are responsible for:  The practices and concerns for testers to have their work become an asset to the team and not a drag in this exciting new world of DevOps.

This is part 2 of a 2-part series on Continuous Testing. Part 1 was focused on the rationale, business goals, culture change, and overview with some specific testing tasks. Part 2 is a deeper dive into the specific ideas, practices, and tasks of test teams that need to be updated or changed to be successful in this next phase of product development. This focuses on test strategy and Automation goals.

We gave a quick overview of the Continuous Testing Process in Part 1:

  1. Start with an automated smoke test. Move these into CI build process tool.
  2. Build bigger regression suites. Move these into CI build process tool.
  3. Grow in levels of awesomeness of CI; Run smoke and/or regression on multiple VMs.
  4. Easy and effective reporting back to the organization.
  5. Use containers and/or virtualization for data and full production-like environments.
  6. Distribute automated tests into different suites with varying goals in different environments. Use VMs for various environments to grow Automation, coverage, speed, monitoring, and feedback to the team.

As we get into this deeper discussion on Continuous Testing and Test Automation strategy, there is an additional idea to remember: DevOps is about delivering value to the customer when the business wants and needs it, not when Dev or Test or Ops gets to it.

These topics are very closely related.

  • Continuous Testing Strategy
  • Rethink Your Automation
  • Change Your Understanding of Regression
  • Embracing Lean and Mean Automated Regression Suites

Now, let’s go into more detail.

Continuous Testing Strategy

The first step towards an effective Continuous Testing strategy is understanding what exactly Continuous Testing (CT) is. By definition, Continuous Testing is a Software Testing practice that typically involves a process of testing early, testing often, testing everywhere, and automating your testing. It emphasizes on evaluating quality at every step of the Continuous Integration and Delivery process (CI/CD).

Continuous Testing is not merely testing continuously–it’s much more intelligent than that! A very common mistake is to run and rerun the same “full regression suite” on different environments and think that is enough to be “doing Continuous Testing.” 

Instead, a key part of developing your Continuous Testing strategy is thinking about testing in different environments, including which environments you’ll need to test on and why you’ll test on that environment–you need to think of what and why we test on different environments. In Agile/Scrum, Dev and Test teams “progressed” through these environments. Now, in DevOps/Continuous Testing, you might be running your Automation on many or all of these environments concurrently. For example:

  • What tests need to be run on the build environment? 
  • What tests need to be run on the test environment? 
  • What automated tests get run against, for example, VMs with various configurations, languages, browsers, mobile devices, etc.? 
  • What gets run in production? 

We also must examine other questions, like, do we understand the results? Are they what we expected or the results we were looking for? Most of all, do they satisfy the client’s needs, expectations, and project requirements?

Many times, senior management judges “Software Testing success” solely based on one metric: cost. However, there are so many other relevant metrics to take into account when analyzing the success of a testing project! Management should stop seeing simply the cost of Software Testing, and instead see the value that Software Testing brings to the software development lifecycle (SDLC). When measuring the success of a testing initiative, here are some other metrics to take into account:

  • The breadth of test coverage
  • Test reusability
  • Value of the data reported by Software Testing
  • The quality of the software under test

To learn more about why these specific metrics are beneficial, check out our post on the LogiGear Blog, How to Measure Testing Success.

Rethink Test Automation

Continuous Testing relies on… Continuous Testing! Thus, everything needs to be automated–or better yet, everything that can be automated needs to be. This concept is the single key to successful Continuous Testing.

What tests need to be automated, what tests do not, how often they should be run and on which environments, how fast they are to run, the possibility that the test will “fail,” the size of tests and their maintenance… all of these factors should impact what and how things will get automated. 

Many people talk about Automation in DevOps. Often, though, they do not mean Test Automation. They may be referring to Task Automation by the Ops team. Automating building environments, provisioning databases, populating user lists… These common tasks for many IT/Ops teams can now be scheduled, automated, and run significantly faster due to the flood of new tools focused on Ops and the “Infrastructure as Code” movement.

DevOps demands an entire re-thinking of your Automation program–both Test and Task Automation. For example, it is common to shrink your “full regression” to run faster and in more environments. Shrinking automation suites is an uncommon goal for many test teams, but one that finds itself incredibly useful and necessary in a CT implementation.

Change Your Understanding of Regression

As I stated above, the idea of CT being the rerunning of your full regression suite repeatedly is a common misconception. CT does involve regression; however, in order to be effective and aligned with the goals of both CT and Automation, we’re going to need to rethink and refactor our regression suite(s) from the traditional ideas of regression.

Regression testing is often intertwined with the idea of “defect priority,” which ranges from priority 1 (P1), where the defect needs to be fixed within 24 hours, to priority 4 (P4), where there is definitely a defect, but it does not need to be fixed to match the “exit” criteria. Regression is often only running P1 or P2 tests, where lower-priority tests are of less importance, not automated, and not re-run. But, regression may only be the user story validation tests that were documented and automated along the way based on how much time the Automation Engineers had to automate whatever tests they did rather than focus on long-lasting, confidence-building, bug-finding regression suites. The fact is, regression means different things to different people. This is particularly true where “automate everything” is a common mantra in Continuous Testing. There is a speed to Continuous Delivery and Continuous Deployment that leaves no room for “backup plans.” 

Regression testing, by definition, “is the re-running of functional and non-functional tests to ensure that previously developed and tested software still performs after a change. Basically, it can apply to any test you have to run more than once. Developers usually test, test, and re-test as they are building the solution in a sandbox or against a “pilot”’ version of the solution; this ensures that their code is working as expected, the results are what is expected as per the requirements, the output looks reasonable, and the performance is within whatever performance limits have been set for the solution.

In CT, the need for us to be testing more often and in more environments is pretty obvious. But, the tests that we are running have to change. Understanding the purpose of each environment and the monitoring or information needed from that environment is key to selecting which tests get run.  Rerunning the same tests over and over on the same system with the same data against the same build does not help anyone. We know we have a smoke test suite–a build validation suite that is run in the CI process on the Dev or Test environment. We all have a “full regression” suite that we normally run against the Test environment. Here is where we need to begin reformulating.

Embracing Lean and Mean Automated Regression Suites

For some teams, their automated “full regression” suite can take days to run. Many “full regression” suites have gotten bloated and slow. This is not okay. The new description of regression, which borrows largely from Lean Software Development’s principle of Cut Waste, is that regression now has to be “Lean and mean,” fast and furious–not bloated, and not the same old “whatever we automated last time.” 

Continuous Testing suites need to be economical and efficient. Teams will need to redesign, optimize, and even cut the size of their automated suites for Continuous Testing rather than keep them bloated and slow. 

Why? For a few reasons:

  • Tests get old and obsolete
  • Tests need maintenance
  • Tests that often fail or break easily need to be rethought and redesigned to find bugs, or break faster and be less fragile.
  • Receive more Immediate feedback.
  • An efficient set of tests will be run by other people on the team in Continuous Testing.
  • Get the biggest coverage with the smallest number of tests. That is, get the biggest bang for your buck.

You will need various suites of tests. For example, you may have a few “regression” suites:

  • A fast, smoke regression, 
  • A big, full regression,
  • A smaller, “happy path”/main functionality, bug-finding regression suite, a main or core “Lean and mean” regression suite, or a smaller end-to-end transaction/integration style regression suite for the full integration environments. 

Each will be run at various points in the DevOps process in various environments. It’s important to remember, all this testing is in addition to the new feature development testing that happens inside your sprint..Will you run any of these in production? I hope not. The goals of those suites are not what you want in production! An automated suite of tests you run in production must be extra careful to maintain the integrity of the production environment and users while monitoring system behavior and availability. So, you may want to add on to the above list of “regression suites” a smaller set of whatever you choose to run against production for monitoring.

Testing in Production

For all the years I have been in testing, there has always been a strict rule: No testing in production. The last thing you want is to be messing around on a live system/environment with customers and then accidentally do something to muck up the system. I fully support this notion and strictly adhere to duplicating the data and solution on a sandbox or pilot server so we can play with whatever needs to be played with. It may result in a need to fix or re-work something in production at some point if necessary, but right is right and the customer and end-users need a solution that works and that provides the data output and performance expected-sometimes no matter the cost (although change orders–if approved–can certainly make this less painful).


Developing a test strategy for Continuous Testing is a full reexamination of what gets tested where. This with the Lean Software development (LSD) idea of Quality at Every Step is the foundation of a Continuous Testing strategy. It involves reexamining what you have been calling “regression” and developing multiple Lean and mean suites for immediate feedback.

It also means rethinking what and how you automate. Old style automation, even small isolated automated tests, need to be reexamined and replaced by efficient, low-level tests as well as longer, customer-focused transaction workflows. Test early, test often is the best solution, and Automated Testing–to whatever degree possible–can work well in the delivery team’s favor to remove the human error in the process and greatly speed up the testing process and timeline to production readiness we are all focusing on.

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

Aligning the Dev and Ops Teams DevOps as a philosophy has had as its centerpiece the principle that Dev and Ops teams need to align better. This is a people and organizational principle, not a process centric principle. To me this is more important when adopting DevOps than any other capability or tool. My last post ...
For this month’s book review, I read Continuous Testing for DevOps Professionals: A Practical Guide from Industry Experts, by various authors and edited by Eran Kinsbruner. The book is divided into 4 sections: Fundamentals of Continuous Testing, Continuous Testing for Web Apps, Continuous Testing for Mobile Apps, and The Future of Continuous Testing. The Fundamentals ...
In this article, I share some of my experiences and observations on training teams, mainly in corporate settings. The operative word here is “team”, not “individual”. When training teams or groups in an organization, many of the considerations and benefits are different than those for the individual. We’ll examine those differences and I will share successful solutions. ...
Run your TestArchitect API and headless browser tests inside Docker containers as easy as flipping a switch Docker is a virtualization platform enabling you to create containers – mini virtual machines— which have their own predefined environment, including file system, libraries and settings. Best of all, these light-weight images eat up only a few megabytes ...
LogiGear Magazine – December 2013 – Cloud Testing
LogiGear Magazine June Issue 2020: Transform Your SDLC With Continuous Testing
This post is part of the Pride & Paradev series With continuous deployment, it is common to release new software into production multiple times a day. A regression test suite, no matter how well designed, may still take over 10 minutes to run, which can lead to bottlenecks in releasing changes to production. So, do ...
Introduction Everything changes. It’s the only constant. The landscape of software testing is undergoing a fast and dramatic change driven by societal, business and technology changes that have placed software everywhere. Computing is ubiquitous. There is hardly anything today that doesn’t contain a CPU or information transmission capability, with software to drive it. From smart toasters ...
Do you want to speed up your automated tests by a factor of 10 and deploy your application continuously? In this article we share how the JIRA development team at Atlassian has accomplished this using Stages in Bamboo. Stages have allowed the JIRA Development team to take a week’s worth of testing and condense it ...
The book is an incredibly effective and valuable guide that details the risks that arise when deploying cloud solutions. More importantly, it provides details on how to test cloud services, to ensure that the proposed cloud service will work as described. It is a great start to the topic. The 6 chapters detail a paradigm ...
If you haven’t already caught a glimpse, we announced in January the Testing in DevOps and Continuous Testing videos series, available on YouTube!
LogiGear Magazine – February 2013 – The Rapidly Changing Software Testing Landscape

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news