Risks of Not Integrating QA Into DevOps

Automated Testing is a huge part of DevOps, but without human-performed quality assurance testing, you’re increasing the risk of  lower-quality software making it into production. 

Automated Testing is an essential DevOps practice to increase organizations’ release cadence and code quality. But there are definitely limits to only using Automated Testing. Without human quality assurance (QA) testing, software is released without ever taking the end-user experience into account. This all but ensures lower-quality software with a poor end-user experience goes to production.

Simply, Automation is a huge part of DevOps, but it shouldn’t be confused with eliminating all manual processes. Conflating the 2 can cause otherwise beneficial DevOps practices to do more harm than good. Here, we explain the top 3 risks organizations face if human QA is not integrated with DevOps.

1. Lack of Human Intervention Allows for Errors to Slip Through Cracks

Automated Testing has greatly improved both the speed and code quality associated with builds and releases. But machines aren’t quite human (yet…), and automated delivery practices cannot always grasp some human aspects of a project. At this time, customer experience cannot (by its very nature) be solely replicated by Automation.

Automated Testing is a type of ‘functional testing.’ Functional testing looks to ensure that defined requirements are properly satisfied by the software. There are 8 different types of functional testing that a build should undergo. Some of these tests are automated, while others require manual intervention. When QA teams are not involved in the testing process and integrated into a DevOps culture, builds can get pushed live with obvious user experience failures.

Just because a code passes Automated Testing doesn’t mean its experience does.

In the same way that a basic spell checker wouldn’t catch the error when talking about “Pear Harbor” (when you meant to type “Pearl Harbor”), many automated functional tests may miss what are obvious user experience failures to a human user. Even large-scale, well-established enterprises sometimes struggle to ensure quality in their builds when humans and machines fail to collaborate on QA testing.

The Impact

Failing to properly test software before it is live causes more time and money to fix the problem after the build has been released.

In the mid-2000s, Toyota drivers reported their cars accelerating without them even touching the pedal, causing several accidents and a recall of millions of affected vehicles. These errors within the cars’ installed software caused stock prices to drop and drivers to migrate towards other car brands.

Builds pushed live with user errors that would have been obvious to a human tester drive away current and potential customers due to a negative experience for the end-user. Toyota could have avoided a massive recall and saved their current and future customers from moving towards alternative manufacturers.

The Solution

In order to ensure only the highest-quality builds are being released, organizations should perform manual QA testing alongside automatic testing. The “right” Manual Testing will vary from organization to organization and from build to build. Organizations will need to work to determine what testing is appropriate for them and where the balance between automatic and Manual Testing lies.

Take care to effectively plan, define, and document testing. This helps increase communication between teams and ensures efficiency and efficacy in testing.

An effective plan includes creating a Quality Management Plan, Test Strategy, and Test Case. This chart from AltexSoft helps break down this plan (Figure 1).

Ensuring that QA is a priority when testing software can help organizations reduce errors in their build, saving them time, money, and the loss of customers.

Figure 1—This chart from AltexSoft breaks down planning your QA testing journey

2. Work Silos

“Siloing” refers to the (real or artificial) separation that forms between workers or teams when collaboration isn’t required. While DevOps works to increase communication and collaboration across all teams within an organization, sometimes organizations focus solely on increasing communication between the Development and Operations teams, forgetting about the other teams in the organization. Not fully integrating QA into your DevOps culture permits silos to exist, further breaking down communication between Development/Operations and QA.

The Impact

People who don’t talk to one another on collaborative projects don’t produce excellent work. Ineffective communication and collaboration can lead to assumptions that go to production and ultimately impact end-users.

For example, the Development and Operations teams may assume that QA’s dedicated job is to find and fix any problems in their code. They may be less careful, therefore, checking and testing their code before sending it to QA, since that’s “not their job.”  QA is then bombarded with lower-quality software, which increases testing times, lengthens feedback loops, and significantly slows release cycles.

While this is an extreme example, it represents the problems of the poor communication and negative business KPIs that silos produce.

The Solution

The best solution to reduce the QA silo is to integrate QA into your organization’s DevOps culture. Start by asking questions about the current process such as:

  • How does QA get code and changes from development?
  • If issues are found, how does QA communicate this information to development? (Do they share it at all?)
  • After code passes QA, how does it pass to the Operations team? Is this automated or in-person?

There are several best practices for integrating QA into a DevOps culture, which include:

  1. Integrate Testing teams into Technical Teams, which automatically allows QA to focus on the appropriate human tests and to move beyond manual functional testing.
  2. Incentivize excellent quality (and therefore all teams prioritizing QA) by adjusting individual and team KPIs to include QA. This will help strengthen necessary behavior and encourage a cultural shift.
  3. Facilitate and encourage communication and collaboration between development, operations, and QA in order to optimize their efforts. Remember: DevOps is a cultural shift as much as a technological one!

Reducing—or better yet, eliminating—silos within the workplace will help organizations consistently produce high-quality software, faster.

3. Undefined Quality Expectations

Without QA being integrated into DevOps, the end-user experience remains undefined and walled off from development and operations. The result is often that these teams want their software to pass automated tests but may not consider the full experience. In fact, if silos are extreme enough, lack of end-to-end visibility may mean these teams have no concept of the full end-user experience.

The Impact

When quality expectations are not fully defined, users are not receiving the highest quality software possible. When users experience frustrating software, their negative experience will drive them away from that software, costing companies valuable market share. Additionally, SDLC teams will have to spend more time and money fixing software after its release.

The Solution

Development, operations, and QA should work together to define quality expectations, and management should make these conversations a priority. Quality expectations will look different from organization to organization, but every organization should define the standards of an excellent software company. Teams should also identify metrics to assess whether software meets quality standards. This serves the dual purpose of creating measurable data as well as presenting an opportunity to break down silos and encourage cross-team communication.

Examples of metrics include:

  • Total number of test cases
  • Number of test cases passed/failed
  • Number of defects found/accepted/rejected
  • Number of critical defects

Having well-defined quality standards ensures that development and operations are keeping the end-user experience in mind. Additionally, having minimum-acceptable metric standards helps QA ensure only the highest quality software is going out that will meet end-user needs. While having defined standards and metrics is important, QA must also be aware and keep an eye out for any edge-cases that may occur.

Don’t Risk Your Competitive Edge

Integrating QA into DevOps allows organizations to reduce silos and release higher-quality software, faster. For more information on integrating QA into DevOps, check out “The Role of QA in DevOps” and “5 QA Best Practices for DevOps.”

This article is a republication and was originally published on Inedo.com.

The Inedo Team
As “the tech behind the tech,” Inedo’s products provide Windows-primary DevOps solutions to organizations of any size and in any industry. Inedo’s products—BuildMaster, ProGet, and Otter—emphasize strong visualization of process, ease-of-use for Developers of all skill levels, and building on the tools and processes you already have in place.

The Related Post

Cross-Browser Testing is an integral part of the Software Testing world today. When we need to test the functionality of a website or web application, we need to do so on multiple browsers for a multitude of reasons.
All too often, software development organizations look at automating software testing as a means of executing existing test cases faster. Very frequently there is no strategic or methodological underpinning to such an effort. The approach is one of running test cases faster is better, which will help to deliver software faster. Even in organizations that ...
One of the basic challenges with test automation is adoption. I can’t tell you how many times I’ve cataloged licenses for a company and found out they already have many different automation software packages, none of which is being used. Traditionally I’ve been told that is because the tools don’t work and that the teams ...
The challenges with any automation effort is to know your capability. I’ve seen too many automation efforts begin and end with a tool decision. Generally these tools are very complex pieces of software that do many more things then we would ever use in our normal everyday testing. It even adds more misery to the ...
LogiGear Magazine – September 2010
The guide for CUI Automated Testing strategies, including chatbot testing and voice app testing. In the Software Testing industry, trends come and go that shape the future of testing. From Automation in Agile, to the DevOps era we are now in, trends are what evolve and improve our testing processes and ideologies. Currently, many researchers ...
Take 5 and test your knowledge on the Software Testing essentials covered in our March 2019 issue. Good luck!
There is no one recipe to make big testing a big success. It takes planning and careful execution of the various aspects, like test design, infrastructure and organization – a mix that can be different for each situation in which you may find yourself. In writing about big testing, the first question that comes up ...
Test Strategy A test strategy describes how the test effort will reach the quality goals set out by the development team. Sometimes called the test approach, test strategy includes, among other things, the testing objective, methods and techniques of testing and the testing environment.
I got some comments on my post “Test Everything all the Time” — most notably people commenting that it’s impossible to test “everything”. I can’t agree more. The intention of the post was to make the point that we need to be able to test “everything we can” all the time. That is, you should ...
“Happy About Global Software Test Automation: A Discussion of Software Testing for Executives” Author: Hung Q. Nguyen, Michael Hackett, and Brent K. Whitlock Publisher: Happy About (August 1, 2006) Finally, a testing book for executives!, November 17, 2006 By Scott Barber “Chief Technologist, PerfTestPlus” Happy About Global Software Test Automation: A Discussion of Software Testing ...
Introduction In many of the Test Automation projects that we are involved with using our Action-Based Testing methodology, management has expressed a need to relate tests and test results to system requirements. The underlying thought is that automation will create extra possibilities to control the level of compliance to requirements of the system under test. ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe