This survey takes an in-depth look at teams that practice DevOps and compares it to teams that don’t practice DevOps.
For 2017, LogiGear is conducting a 4-part survey to assess the state of the software testing practice as it stands today.
This is a 4-part series to mirror LogiGear Magazine’s issues this year.
1. Test Essentials (Back to Basics) – Topics include test plans, test cases, attitudes about quality and testing.
2. DevOps – Topics include Continuous Delivery, training, tools, development lifecycle and IT/Ops integration.
3. Automation – Topics include tools, attitudes, skills and use.
4. Staffing, Outsourcing, Training, Leading and Managing – Topics include training, staffing, management, outsourcing and job satisfaction.
Survey 2, announced in March, is now complete. Starting on the next page are the results and my comments on them. Survey 3 is now open and collecting results. The current survey is on Automation. I encourage you to pass these surveys along to your Test/QA friends to get the widest set of responses and opinions.
This survey on DevOps was unique in that there were 2 separate surveys to choose from and they were predicated on the respondents’ business situations. One survey was for teams that are currently doing Continuous Delivery/Continuous Testing/DevOps practices, and the other was for teams that aren’t. Each survey had ten questions. To compare and contrast both survey groups, we asked 5 of the same questions to both groups. Delving deeper in the practice, we asked the survey group that does practice DevOps 5 unique questions towards that field to understand how they’re doing. To better understand why those that aren’t doing DevOps, we asked 5 unique questions to better determine their business reasoning.
DevOps is a journey. It means a lot of things to different people.
We can’t simply turn a key one day and “do” DevOps. We learned this through the problem era of implementing Agile/Scrum with mainly ScrumBut practices.
There are two fully differentiating aspects here when it comes to moving to DevOps. One is that, moving to DevOps involves groups that may not naturally work well together or have the same goals. The second is that there will have to be financial commitments to the tool chain (often a very large commitment to one tool or another). Organizations cannot fall into DevOps without a lot of planning, training, culture change, etc.
What we want to do in this survey is to take a snapshot of where we are as a software testing community inside product development.
Are Your DevOps in Good Shape?
Right off the bat, as you can see in the first question, it’s encouraging that most people know what is ahead in DevOps even if they have not adopted the practices yet.
However, in question #2, these results are a mixed bag. By now— after over a dozen years of Agile/Scrum being mainstreamed in software development— only 1/3 say their practices are in good shape. 67% responded “Sometimes” or “No.” Even without moving to DevOps, this is disappointing. There is so much information and so many consultants out in the world today who can help improve practices and communication that will benefit the entire organization. DevOps practices rely on so much more on collaboration between teams, including agreement on when an item is done, with no technical debt allowed. Work must always be “release-ready” to make it into the deployment pipeline. Technical debt goes against this idea.
Delivering Your Product Efficiently
Getting value to users faster, when the business decides, not when Dev teams decide, is the foundation of being more predictable in Agile and more so using DevOps. It’s good to see in question 4 that more than half of respondents say getting product/value to users is predictable. About the same percentages responded “slow”/”very slow as fast”/”very fast,” with a sizable group choosing “unpredictable.” The responses are mixed. This could be the epitome of why so many teams are being pushed to improve and streamline their practices to be more responsive to the business and customers. Product moves fast today. Being slow hurts your business.
Talking about Your Ops/IT Teams
Question #10 gives us many ideas. First, can you see the absolute need to have Ops/IT more involved in every part of development? Clearly, environments and data are very often problems for test teams and will slow down or stop any deployment pipeline, should your team be moving in that direction.
Second, even if your organization is not moving to DevOps yet, why, in 2017, are there so many environment and data problems? I see this in my consulting work far, far too often. Many teams “find” bugs that are not really bugs but are due to testing on weird, wrong, incomplete environments with dubious data. This has to stop. It hurts so many teams.
The Tool Side of DevOps
These two questions are not the same, but are related. I asked each group about tool use.
The surprise result here is that for both groups tool use is far too low. For both groups, as expected, the most important tool for testers is their test automation tool. For the No group, only 24% are using the tool. Oddly, for the Yes group, the result was only 20%. This number could be 80% and still be considered low. There are many other possible reasons for such low numbers. For example, many of the respondents could be test engineers or subject matter experts (test engineers who design but do not implement automated tests). This could also be the same for any teams doing BDD with a tool like Cucumber. 100% could be designing automated tests, but a much smaller number could be interacting with the tool.
Data and Environment Problems: A Lot vs. Some
Of all the questions in both surveys, this was the most surprising answer; the comparison of these two exacerbates the problem. “Do you have environment or data problems?” Many times in my consulting work I see test teams struggle with environment and data problems. Implementing DevOps must fix these problems. The fact that so many teams, in 2017, still suffer from these issues is a problem. With all the possible solutions— cloud, EaaS, IaaS, etc.—it’s unacceptable that these problems are so common. Also, by bringing Ops/IT earlier into our process— we should be fixing these issues since they are blocking or impeding the pipeline. These issues are constraining the flow.
The group not doing DevOps has fewer problems in this area. The group doing DevOps has more. That is counter-intuitive. There is one possible reason that would explain all this— with teams doing DevOps, they are more than likely using a release and/or configuration management tool, or suite of tools. Existing practices, automated test suites, and tools now have to fit inside that tool, and many teams do have problems transitioning to have a tool automatically run their tests in various platforms/environments. This change will initially create environment and data problems for DevOps teams to solve. In the long term, environment and data problems need to disappear— this is an identified bottleneck to deployment.
A Herculean Effort on Both Sides
The effort required to get issues fixed and services restored seem to be about the same for DevOps and non-DevOps teams. There are twice as many respondents saying it is “simple and no extra effort” for DevOps teams.
Luckily, for both sets of responses, the effort is rarely Herculean!
Team Awareness is Key
These two are not the same but related. Here we do see a difference in awareness, collaboration, and information on all testing activities across the SDLC. This is the expected answer. With the focus on collaboration, feedback, and shared responsibility for quality, DevOps teams must be aware of all the testing activities as well as their results.
Similarly, (even if you are not doing DevOps) you can’t let anyone on your team off the hook for not knowing and leveraging all the quality practices. In today’s faster environment, no one can afford redundancy or gaps. Knowledge is the key.
Smart Automation vs. Pressured Automation
These results are as expected. In DevOps teams, a common complaint as well as a requirement is increased automation! This leads to pressure. I would advise teams suffering from too much pressure to do smarter automation as opposed to more and more— and communicate this idea.
Furthermore, the following questions are as expected and also very interesting to me. Teams rating their own automation suites is revealing. The automated suites seem to be better, easier, and more efficient in DevOps teams than in non-DevOps teams— as expected.
It is important to point out one particular result: 37% of teams not doing DevOps have no significant automation. This is a problem our industry still faces. This result was apparent from the last survey. There are still many, many teams—for various reasons—not doing any test automation at all. This is a problem for each of those individual teams, but also a mark on our industry as a whole. This is still far too common.
Training, Training, Training
Before analyzing the numbers here, let’s think again about what DevOps is. At its most basic level, DevOps is significantly increased and improved communication as well as collaboration between Dev, QA, and Ops/IT.
Only a tiny percentage has been trained in a group where questions can be openly discussed, understandings can be agreed upon among many people, and information can be broadcast or spread out. It is not a good start for over one-third to not be trained! Also, as good as it is for more than half to have done so on their own, they are doing this alone, rather than across teams where they can be exposed to different ideas and descriptions of practices.
I had hoped we learned our lessons from the slow difficult adoption of Agile, that DevOps starts with culture shifts and greater collaboration. This is done so much easier and faster through training than buying a tool and dictating to individual silos of people how they should use the tool, rather than what problems they are trying to solve.
Positive Results for Testing
This response is what I expected. Over 60% of respondents are executing more tests. I would be interested to see the reasons for the 17% responding “less tests.” It would be a pretty sophisticated set of practices and tools that narrow testing down to smaller chunks of affected code, rather than widespread testing, thereby executing less tests. Bravo and good luck if they are!
Developers and Unit Testing
Great news here that more than half responded that developers were doing “more” and “a lot more” unit testing. Great news.
The fact that 11% do not know, is a problem. DevOps = greater collaboration & communication. Not knowing is a problem. Also, I am not sure how teams can be doing CI/CD practices without formal, automated unit test suites.
Faster, Better Delivery
This is the obvious result. Good for that! 73% responded they are delivering product/value to their customers faster or much faster. This is one of the main reasons for DevOps. Good for you!
On the Yes and No surveys there were a few identical questions where the goal was to observe how things change or not change by implementing these practices.
Summary
As more teams practice DevOps longer I would hope that the communication, information, training, and collaboration answers improve across the board. There seem to be too many gaps in simply talking to each other and sharing information. For all of us as well, data and environments are already identified constraints. These bottlenecks must be erased. And as always, as an industry group, we must automate more and smartly.
We have reached the end of this installment. As promised, we’ll share the full review, with thoughts and opinions in our upcoming ebook. In the meantime, please take our Test Automation Survey, or peruse other articles in this issue. Don’t forget to share the survey with your network! The more recipients we get, the better reflection we get on the current state of the industry!
