Risks of Not Integrating QA Into DevOps

Automated Testing is a huge part of DevOps, but without human-performed quality assurance testing, you’re increasing the risk of  lower-quality software making it into production. 

Automated Testing is an essential DevOps practice to increase organizations’ release cadence and code quality. But there are definitely limits to only using Automated Testing. Without human quality assurance (QA) testing, software is released without ever taking the end-user experience into account. This all but ensures lower-quality software with a poor end-user experience goes to production.

Simply, Automation is a huge part of DevOps, but it shouldn’t be confused with eliminating all manual processes. Conflating the 2 can cause otherwise beneficial DevOps practices to do more harm than good. Here, we explain the top 3 risks organizations face if human QA is not integrated with DevOps.

1. Lack of Human Intervention Allows for Errors to Slip Through Cracks

Automated Testing has greatly improved both the speed and code quality associated with builds and releases. But machines aren’t quite human (yet…), and automated delivery practices cannot always grasp some human aspects of a project. At this time, customer experience cannot (by its very nature) be solely replicated by Automation.

Automated Testing is a type of ‘functional testing.’ Functional testing looks to ensure that defined requirements are properly satisfied by the software. There are 8 different types of functional testing that a build should undergo. Some of these tests are automated, while others require manual intervention. When QA teams are not involved in the testing process and integrated into a DevOps culture, builds can get pushed live with obvious user experience failures.

Just because a code passes Automated Testing doesn’t mean its experience does.

In the same way that a basic spell checker wouldn’t catch the error when talking about “Pear Harbor” (when you meant to type “Pearl Harbor”), many automated functional tests may miss what are obvious user experience failures to a human user. Even large-scale, well-established enterprises sometimes struggle to ensure quality in their builds when humans and machines fail to collaborate on QA testing.

The Impact

Failing to properly test software before it is live causes more time and money to fix the problem after the build has been released.

In the mid-2000s, Toyota drivers reported their cars accelerating without them even touching the pedal, causing several accidents and a recall of millions of affected vehicles. These errors within the cars’ installed software caused stock prices to drop and drivers to migrate towards other car brands.

Builds pushed live with user errors that would have been obvious to a human tester drive away current and potential customers due to a negative experience for the end-user. Toyota could have avoided a massive recall and saved their current and future customers from moving towards alternative manufacturers.

The Solution

In order to ensure only the highest-quality builds are being released, organizations should perform manual QA testing alongside automatic testing. The “right” Manual Testing will vary from organization to organization and from build to build. Organizations will need to work to determine what testing is appropriate for them and where the balance between automatic and Manual Testing lies.

Take care to effectively plan, define, and document testing. This helps increase communication between teams and ensures efficiency and efficacy in testing.

An effective plan includes creating a Quality Management Plan, Test Strategy, and Test Case. This chart from AltexSoft helps break down this plan (Figure 1).

Ensuring that QA is a priority when testing software can help organizations reduce errors in their build, saving them time, money, and the loss of customers.

Figure 1—This chart from AltexSoft breaks down planning your QA testing journey

2. Work Silos

“Siloing” refers to the (real or artificial) separation that forms between workers or teams when collaboration isn’t required. While DevOps works to increase communication and collaboration across all teams within an organization, sometimes organizations focus solely on increasing communication between the Development and Operations teams, forgetting about the other teams in the organization. Not fully integrating QA into your DevOps culture permits silos to exist, further breaking down communication between Development/Operations and QA.

The Impact

People who don’t talk to one another on collaborative projects don’t produce excellent work. Ineffective communication and collaboration can lead to assumptions that go to production and ultimately impact end-users.

For example, the Development and Operations teams may assume that QA’s dedicated job is to find and fix any problems in their code. They may be less careful, therefore, checking and testing their code before sending it to QA, since that’s “not their job.”  QA is then bombarded with lower-quality software, which increases testing times, lengthens feedback loops, and significantly slows release cycles.

While this is an extreme example, it represents the problems of the poor communication and negative business KPIs that silos produce.

The Solution

The best solution to reduce the QA silo is to integrate QA into your organization’s DevOps culture. Start by asking questions about the current process such as:

  • How does QA get code and changes from development?
  • If issues are found, how does QA communicate this information to development? (Do they share it at all?)
  • After code passes QA, how does it pass to the Operations team? Is this automated or in-person?

There are several best practices for integrating QA into a DevOps culture, which include:

  1. Integrate Testing teams into Technical Teams, which automatically allows QA to focus on the appropriate human tests and to move beyond manual functional testing.
  2. Incentivize excellent quality (and therefore all teams prioritizing QA) by adjusting individual and team KPIs to include QA. This will help strengthen necessary behavior and encourage a cultural shift.
  3. Facilitate and encourage communication and collaboration between development, operations, and QA in order to optimize their efforts. Remember: DevOps is a cultural shift as much as a technological one!

Reducing—or better yet, eliminating—silos within the workplace will help organizations consistently produce high-quality software, faster.

3. Undefined Quality Expectations

Without QA being integrated into DevOps, the end-user experience remains undefined and walled off from development and operations. The result is often that these teams want their software to pass automated tests but may not consider the full experience. In fact, if silos are extreme enough, lack of end-to-end visibility may mean these teams have no concept of the full end-user experience.

The Impact

When quality expectations are not fully defined, users are not receiving the highest quality software possible. When users experience frustrating software, their negative experience will drive them away from that software, costing companies valuable market share. Additionally, SDLC teams will have to spend more time and money fixing software after its release.

The Solution

Development, operations, and QA should work together to define quality expectations, and management should make these conversations a priority. Quality expectations will look different from organization to organization, but every organization should define the standards of an excellent software company. Teams should also identify metrics to assess whether software meets quality standards. This serves the dual purpose of creating measurable data as well as presenting an opportunity to break down silos and encourage cross-team communication.

Examples of metrics include:

  • Total number of test cases
  • Number of test cases passed/failed
  • Number of defects found/accepted/rejected
  • Number of critical defects

Having well-defined quality standards ensures that development and operations are keeping the end-user experience in mind. Additionally, having minimum-acceptable metric standards helps QA ensure only the highest quality software is going out that will meet end-user needs. While having defined standards and metrics is important, QA must also be aware and keep an eye out for any edge-cases that may occur.

Don’t Risk Your Competitive Edge

Integrating QA into DevOps allows organizations to reduce silos and release higher-quality software, faster. For more information on integrating QA into DevOps, check out “The Role of QA in DevOps” and “5 QA Best Practices for DevOps.”

This article is a republication and was originally published on Inedo.com.

The Inedo Team
As “the tech behind the tech,” Inedo’s products provide Windows-primary DevOps solutions to organizations of any size and in any industry. Inedo’s products—BuildMaster, ProGet, and Otter—emphasize strong visualization of process, ease-of-use for Developers of all skill levels, and building on the tools and processes you already have in place.

The Related Post

In order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.
Mobile usage today is not just a trend but it is an essential shift in how people communicate with each other, interact with the world, and do business. According to a ComScore, in 2014 the number of mobile users surpassed the number of computer users and is showing strong growth over time, towards some point in ...
LogiGear Magazine, December 2015: Test Automation
The huge range of mobile devices used to browse the web now means testing a mobile website before delivery is critical.
Even the highest quality organizations have tradeoffs when it comes to their testing coverage. In Japan, Europe, and the United States, automotive manufacturers are aiming to enhance automotive functions by using software; in Japan in particular, Toyota, Nissan, Honda, Mazda, and Subaru are all adding endless amounts of software to their vehicles in the form ...
September Issue 2019: Advancing Automation
One of my current responsibilities is to find ways to automate, as much as practical, the ‘testing’ of the user experience (UX) for complex web-based applications. In my view, full test automation of UX is impractical and probably unwise; however, we can use automation to find potential UX problems, or undesirable effects, even in rich, ...
LogiGear Magazine January Trends Issue 2017
The 12 Do’s and Don’ts of Test Automation When I started my career as a Software Tester a decade ago, Test Automation was viewed with some skepticism.
< Michael Hackett sat down with EA’s Stephen Copp to discuss the world of integrated test platforms.
How to do UI test automation with the fewest headaches I’m currently interviewing lots of teams that have implemented acceptance testing for my new book. A majority of those interviewed so far have at some point shot themselves in the foot with UI test automation. After speaking to several people who are about to do ...
Investing in Test Automation training will increase your team’s productivity. The availability of reliable jobs in a competitive US market seems to be constantly embattled with competition and replacements of artificial intelligence (AI). In 2016, Foxconn replaced 60,000 employees with robots. However, the growth of Test Automation as an occupation has highlighted an intriguing option ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe