Continuous Testing… what is it? When we first decided to do a magazine issue dedicated to the DevOps practice of Continuous Testing, I joked with someone: “It’s about testing continuously.”
And their reply was: “Yeah. What else would it be?”
I was joking, but clearly the joke didn’t land. Continuous Testing is about testing continuously, but it’s not that simple. The DevOps practice of Continuous Testing and what that looks like for your organization is going to be reliant on a few things, but let’s first talk about DevOps.
Remember, DevOps is, at its heart, about getting Development, Operations, and Testing all talking and collaborating early in the development process–namely, from the start of a project. In some cases, DevOps has morphed into choosing a toolset for Continuous Delivery. As important as that is, a toolset does not automatically mean you are doing “DevOps.” With more and earlier collaboration about testing and permanently breaking down the Silos there will be more testing, earlier in development.
Bringing the focus back to Continuous Testing, there are 2 cultural principles that have a greater influence on what Continuous Testing means: (1) shift-left, and (2) quality at every step. Shift-left is the development ideology that first gained attention in Agile by taking the testing stage, which was historically at the very end of the SDLC, and moving it earlier in the SDLC, thus shifting the testing “left” on the pipeline. This leads into quality at every step because by testing earlier (and more often) in the SDLC, you’re ensuring that after each phase, the quality of the software is good enough to be promoted to the next phase _ thus, quality at every step [of the SDLC].
First, you have to have a smooth, easy continuous integration/CI practice. This is build validation, meaning you get a build and run tests to see that you have a good build. First, you run the unit tests; then, run a smoke test consisting of higher-level tests focused more on integration than on individual units. Once the build is qualified, the team can go ahead and do more development and do more testing.
The thing about Continuous Testing, understanding both shift-left and quality at every step, is you do one thing and test it. Then, you do another thing and test that thing. The purpose here is if you break something but you only changed one thing, you know exactly why the break happened. This will make defect isolation and test it. Then, you do another thing and test that thing. The purpose here is if you break something but you only changed one thing, you know exactly why the break happened. This will make defect isolation and reworking pretty quick. In Continuous Testing, your testing is narrow because your change is narrow.
Think about it another way. If you are doing container architecture, one of the main benefits is that if you swap out one container with another container, you only have to test the container you changed out. You don’t have to run a 5-day automated GUI test suite consisting of 800,000 test cases across 5 platforms–you just run a test for the area that changed. You make a change and you test that change. Continuous Testing does not mean you run your full regression suite, again and again, every time you get a new build. You don’t need to test the whole thing.
You may ask, “Michael, are you saying if I change one part of a system, that there is no need to test another part of the system? Can’t one change to break a faraway piece?” To that, I say yes and no: It is true that one change can break another faraway piece; however, we all know that can happen. And that is a risk of speedy delivery and deployment.
But now this brings up the numbers game, and numbers are important.
Let’s say you have 10,000 automated tests… if it takes you only 20 minutes to run those 10,000 test cases, great! Go ahead and run your full regression suite every time you make any change or promote to the next environment if it’s only 20 minutes. But if those 10,000 tests take overnight to run, clearly you can’t run that test suite every time you push another build-out, or after every environment change. If your full regression suite consists of significant maintenance time, significant execution time, and significant cost, then you clearly cannot be continuously running that same full regression suite. It’s important to note that Continuous Testing is not about running tests continuously, but instead about delivering results continuously and quickly.
So, to refer back to my initial failed joke: Yes, Continuous Testing is, at a basic level, about testing continuously. However, there are clearly a lot of moving parts, requirements, and contingencies. There is no defined right or wrong way to Continuous Testing, but instead a bunch of grey areas that are dependent on your organization’s processes and goals–which makes it a really great topic for a 30+ page magazine!
The goal of this issue is to answer one question as thoroughly as we can: What is Continuous Testing? This issue brings together thought leadership from both inside and outside of LogiGear. The cover story, 5 Best Practices for Continuous Testing, was written by LogiGear CTO Hans Buwalda, in which he applies his decades of knowledge regarding test design and Automation to offer 5 tips for organizations who may be struggling with a CT implementation. Our infographic, The Beginner’s Guide to Continuous Testing, is a great resource for those who may be learning about or looking to implement CT for the first time. In the article Continuous Delivery: Everything You Need to Know, LogiGear Product Manager Thuc Nguyen covers “everything you need to know” (ha!) about Continuous Delivery, from definitions to tools choice, to testing in Continuous Delivery. This issue’s Blogger of the Month, Kristin Jackvony, shares her insight in Book Review: Continuous Testing for DevOps Professionals, which is a great guide written by industry experts for those seeking to optimize their CT program. Our strategic business partner Sauce Labs also contributed to this issue with Continuous Testing in the Retail Industry, which explores the growing “need for testing speed” in digital retail marketplaces. This issue also features the final installment of my 4-year Leader’s Pulse series; this final installment is essentially a culmination of my original intent for this series: What is the Value of a Manager’s Role in Modern Tech Organizations? And finally, to wrap it all up, our TestArchitect Team has created a step-by-step guide on How to Integrate The Jenkins CI Tool into TestArchitect.
We hope you enjoy this issue. Happy Testing!
