Continuous Testing, Part 2: Strategy and Automation Goals for Test Teams

DevOps has been described as Agile on Steroids; DevOps has also been described as Agile for Operations/IT. I like both of those descriptions. Many organizations want Development, Test, and Operations teams to move to DevOps now.

DevOps is a big topic, but DevOps is not the focus of this article. We will not be talking about, for example, containers, or Docker, or Puppet, or Infrastructure as code. In this article, we are going to focus on Continuous Testing. I will focus on what test teams are responsible for:  The practices and concerns for testers to have their work become an asset to the team and not a drag in this exciting new world of DevOps.

This is part 2 of a 2-part series on Continuous Testing. Part 1 was focused on the rationale, business goals, culture change, and overview with some specific testing tasks. Part 2 is a deeper dive into the specific ideas, practices, and tasks of test teams that need to be updated or changed to be successful in this next phase of product development. This focuses on test strategy and Automation goals.

We gave a quick overview of the Continuous Testing Process in Part 1:

  1. Start with an automated smoke test. Move these into CI build process tool.
  2. Build bigger regression suites. Move these into CI build process tool.
  3. Grow in levels of awesomeness of CI; Run smoke and/or regression on multiple VMs.
  4. Easy and effective reporting back to the organization.
  5. Use containers and/or virtualization for data and full production-like environments.
  6. Distribute automated tests into different suites with varying goals in different environments. Use VMs for various environments to grow Automation, coverage, speed, monitoring, and feedback to the team.

As we get into this deeper discussion on Continuous Testing and Test Automation strategy, there is an additional idea to remember: DevOps is about delivering value to the customer when the business wants and needs it, not when Dev or Test or Ops gets to it.

These topics are very closely related.

  • Continuous Testing Strategy
  • Rethink Your Automation
  • Change Your Understanding of Regression
  • Embracing Lean and Mean Automated Regression Suites

Now, let’s go into more detail.

Continuous Testing Strategy

The first step towards an effective Continuous Testing strategy is understanding what exactly Continuous Testing (CT) is. By definition, Continuous Testing is a Software Testing practice that typically involves a process of testing early, testing often, testing everywhere, and automating your testing. It emphasizes on evaluating quality at every step of the Continuous Integration and Delivery process (CI/CD).

Continuous Testing is not merely testing continuously–it’s much more intelligent than that! A very common mistake is to run and rerun the same “full regression suite” on different environments and think that is enough to be “doing Continuous Testing.” 

Instead, a key part of developing your Continuous Testing strategy is thinking about testing in different environments, including which environments you’ll need to test on and why you’ll test on that environment–you need to think of what and why we test on different environments. In Agile/Scrum, Dev and Test teams “progressed” through these environments. Now, in DevOps/Continuous Testing, you might be running your Automation on many or all of these environments concurrently. For example:

  • What tests need to be run on the build environment? 
  • What tests need to be run on the test environment? 
  • What automated tests get run against, for example, VMs with various configurations, languages, browsers, mobile devices, etc.? 
  • What gets run in production? 

We also must examine other questions, like, do we understand the results? Are they what we expected or the results we were looking for? Most of all, do they satisfy the client’s needs, expectations, and project requirements?

Many times, senior management judges “Software Testing success” solely based on one metric: cost. However, there are so many other relevant metrics to take into account when analyzing the success of a testing project! Management should stop seeing simply the cost of Software Testing, and instead see the value that Software Testing brings to the software development lifecycle (SDLC). When measuring the success of a testing initiative, here are some other metrics to take into account:

  • The breadth of test coverage
  • Test reusability
  • Value of the data reported by Software Testing
  • The quality of the software under test

To learn more about why these specific metrics are beneficial, check out our post on the LogiGear Blog, How to Measure Testing Success.

Rethink Test Automation

Continuous Testing relies on… Continuous Testing! Thus, everything needs to be automated–or better yet, everything that can be automated needs to be. This concept is the single key to successful Continuous Testing.

What tests need to be automated, what tests do not, how often they should be run and on which environments, how fast they are to run, the possibility that the test will “fail,” the size of tests and their maintenance… all of these factors should impact what and how things will get automated. 

Many people talk about Automation in DevOps. Often, though, they do not mean Test Automation. They may be referring to Task Automation by the Ops team. Automating building environments, provisioning databases, populating user lists… These common tasks for many IT/Ops teams can now be scheduled, automated, and run significantly faster due to the flood of new tools focused on Ops and the “Infrastructure as Code” movement.

DevOps demands an entire re-thinking of your Automation program–both Test and Task Automation. For example, it is common to shrink your “full regression” to run faster and in more environments. Shrinking automation suites is an uncommon goal for many test teams, but one that finds itself incredibly useful and necessary in a CT implementation.

Change Your Understanding of Regression

As I stated above, the idea of CT being the rerunning of your full regression suite repeatedly is a common misconception. CT does involve regression; however, in order to be effective and aligned with the goals of both CT and Automation, we’re going to need to rethink and refactor our regression suite(s) from the traditional ideas of regression.

Regression testing is often intertwined with the idea of “defect priority,” which ranges from priority 1 (P1), where the defect needs to be fixed within 24 hours, to priority 4 (P4), where there is definitely a defect, but it does not need to be fixed to match the “exit” criteria. Regression is often only running P1 or P2 tests, where lower-priority tests are of less importance, not automated, and not re-run. But, regression may only be the user story validation tests that were documented and automated along the way based on how much time the Automation Engineers had to automate whatever tests they did rather than focus on long-lasting, confidence-building, bug-finding regression suites. The fact is, regression means different things to different people. This is particularly true where “automate everything” is a common mantra in Continuous Testing. There is a speed to Continuous Delivery and Continuous Deployment that leaves no room for “backup plans.” 

Regression testing, by definition, “is the re-running of functional and non-functional tests to ensure that previously developed and tested software still performs after a change. Basically, it can apply to any test you have to run more than once. Developers usually test, test, and re-test as they are building the solution in a sandbox or against a “pilot”’ version of the solution; this ensures that their code is working as expected, the results are what is expected as per the requirements, the output looks reasonable, and the performance is within whatever performance limits have been set for the solution.

In CT, the need for us to be testing more often and in more environments is pretty obvious. But, the tests that we are running have to change. Understanding the purpose of each environment and the monitoring or information needed from that environment is key to selecting which tests get run.  Rerunning the same tests over and over on the same system with the same data against the same build does not help anyone. We know we have a smoke test suite–a build validation suite that is run in the CI process on the Dev or Test environment. We all have a “full regression” suite that we normally run against the Test environment. Here is where we need to begin reformulating.

Embracing Lean and Mean Automated Regression Suites

For some teams, their automated “full regression” suite can take days to run. Many “full regression” suites have gotten bloated and slow. This is not okay. The new description of regression, which borrows largely from Lean Software Development’s principle of Cut Waste, is that regression now has to be “Lean and mean,” fast and furious–not bloated, and not the same old “whatever we automated last time.” 

Continuous Testing suites need to be economical and efficient. Teams will need to redesign, optimize, and even cut the size of their automated suites for Continuous Testing rather than keep them bloated and slow. 

Why? For a few reasons:

  • Tests get old and obsolete
  • Tests need maintenance
  • Tests that often fail or break easily need to be rethought and redesigned to find bugs, or break faster and be less fragile.
  • Receive more Immediate feedback.
  • An efficient set of tests will be run by other people on the team in Continuous Testing.
  • Get the biggest coverage with the smallest number of tests. That is, get the biggest bang for your buck.

You will need various suites of tests. For example, you may have a few “regression” suites:

  • A fast, smoke regression, 
  • A big, full regression,
  • A smaller, “happy path”/main functionality, bug-finding regression suite, a main or core “Lean and mean” regression suite, or a smaller end-to-end transaction/integration style regression suite for the full integration environments. 

Each will be run at various points in the DevOps process in various environments. It’s important to remember, all this testing is in addition to the new feature development testing that happens inside your sprint..Will you run any of these in production? I hope not. The goals of those suites are not what you want in production! An automated suite of tests you run in production must be extra careful to maintain the integrity of the production environment and users while monitoring system behavior and availability. So, you may want to add on to the above list of “regression suites” a smaller set of whatever you choose to run against production for monitoring.

Testing in Production

For all the years I have been in testing, there has always been a strict rule: No testing in production. The last thing you want is to be messing around on a live system/environment with customers and then accidentally do something to muck up the system. I fully support this notion and strictly adhere to duplicating the data and solution on a sandbox or pilot server so we can play with whatever needs to be played with. It may result in a need to fix or re-work something in production at some point if necessary, but right is right and the customer and end-users need a solution that works and that provides the data output and performance expected-sometimes no matter the cost (although change orders–if approved–can certainly make this less painful).


Developing a test strategy for Continuous Testing is a full reexamination of what gets tested where. This with the Lean Software development (LSD) idea of Quality at Every Step is the foundation of a Continuous Testing strategy. It involves reexamining what you have been calling “regression” and developing multiple Lean and mean suites for immediate feedback.

It also means rethinking what and how you automate. Old style automation, even small isolated automated tests, need to be reexamined and replaced by efficient, low-level tests as well as longer, customer-focused transaction workflows. Test early, test often is the best solution, and Automated Testing–to whatever degree possible–can work well in the delivery team’s favor to remove the human error in the process and greatly speed up the testing process and timeline to production readiness we are all focusing on.

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

Special considerations that should be applied to an application running in the cloud. Over the last weeks, I have found myself in several rather intense discussions about “cloud testing”: what it is, what it isn’t, and what it means for testing and QA professionals. The major source of confusion in these discussions usually revolves around ...
  LogiGear_Magazine_June 2016_Testing in the New World of DevOps  
How Halliburton leveraged outsourcing to achieve their goals. Organizations are focusing on speed, both in Continuous Integration and rapid deployment as a competitive advantage. Many software development organizations can significantly shorten development cycles by implementing one or a combination of Agile practices, continuous integration & deployment methods, and feature branches. While these frameworks and techniques ...
How to ensure a successful test-driven environment In order to ensure a higher quality product is released in the end, many teams have turned to test-driven development. Under this scenario, quality assurance metrics professionals first create various QA tests, and then software engineers code based on these tests, typically while using a robust enterprise test management ...
Continuous Testing and Continuous Monitoring What is the goal of Continuous Integration? Is it to enable Continuous Delivery of the code developers’ produce out to users? Yes, eventually. But first and foremost it is to enable ongoing test and verification of the code. It is to validate that the code produced and integrated with that ...
From adopting the culture, to implementing Continuous Delivery With the relative newness of DevOps, there are not yet a ton of DevOps books. That’s why we’ve assembled a list of the 7 best DevOps books based on four criteria: the number of ratings from Amazon, the average Amazon rating, number of ratings from GoodReads and the ...
A test team’s job is to report test results, not set or guarantee that you will meet the SLAs. In the rush to cloud services, with everything-as-a-service, you will hear people talking about SLAs. What is this about and what does it have to do with testing? A Service Level Agreement, or SLA, is a ...
Do you want to speed up your automated tests by a factor of 10 and deploy your application continuously? In this article we share how the JIRA development team at Atlassian has accomplished this using Stages in Bamboo. Stages have allowed the JIRA Development team to take a week’s worth of testing and condense it ...
Cloud computing has been the buzzword in the world of Information Technology for quite some time and it is likely to retain that status in the coming years. Cloud computing has been helping business enterprises deliver services faster and cheaper compared to all other existing delivery models. Small and medium business enterprises have changed their ...
Disclaimer: This article was originally published in the LogiGear Magazine, but has been updated for the June 2020 issue. Leading an organization into Continuous Delivery is a daunting task that usually takes software development teams months if not years. This guide covers the main points that organizations need to consider to increase their chance of ...
Over the years we’ve provided an extensive number of articles, videos, and infographics that provide a wealth of knowledge about Continuous Delivery.
DevOps for Test Teams By Michael Hackett Now that Dev Teams have had a little time to settle into the Agile, the new wave of process optimization has arrived. DevOps. DevOps has been described as Agile on Steroids. DevOps has also been described as Agile for Operations/IT. I like both of those descriptions as well ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news