CONTINUOUS TESTING, PART 2: Strategy and Automation Goals for Test Teams

By  Michael Hackett

Now that Dev Teams have had a little time to settle into the Agile, the new wave of process optimization has arrived. DevOps. DevOps has been described as Agile on Steroids. DevOps has also been described as Agile for Operations/IT. I like both of those descriptions as well as some others. Many organizations want Development, Test and Operations teams to move to DevOps now.

DevOps is a big topic. But DevOps is not the focus of this article. We will not be talking about, for example, containers, or Docker, or Puppet or Infrastructure as code. In this article, we are going to focus on Continuous Testing.  I will focus on what Test teams are responsible for–the practices and concerns for testers to have their work become an asset to the team and not a drag in this exciting new world of DevOps.


This is part 2 of a 3 part series on Continuous Testing. Part 1 on the rationale, business goals, culture change and overview with some specific testing tasks. Part 2 is a deeper dive into the specific ideas, practices and tasks of Test teams that need to be updated or changed to be successful in this next phase of product development. This focuses on test strategy, automation goals.  Part 3 will discuss more automation in DevOps/Continuous Testing issues as well as , environment and data problems to solve, virtualization, with more details on testing goals.

We gave a quick overview of the Continuous Testing Process in Part 1.

  1. Start with an automated smoke test. Move these into CI build process tool.
  2. Build bigger regression suites. Move these into CI build process tool.
  3. Grow in levels of awesomeness of CI; Run smoke and/or regression on multiple VMs.
  4. Easy and effective reporting back to the organization.
  5. Use containers and/or virtualization for data and full production like environments.
  6. Distribute automated tests into different suites with varying goals on different environments. Use VMs for various environments to grow automation, coverage, speed, monitoring and feedback to the team.

As we get into this discussion, there is an additional idea to remember.

DevOps is about delivering value to the customer when the business wants and needs it , not when Dev or test or ops gets to it.

Now, let’s go into more detail.

These topics are very closely related.

  • Continuous Testing Strategy
  • Examine and redefine regression.
  • Rethink your automation


Continuous Testing is not merely testing continuously. It’s more intelligent than that. A very common mistake is to run and rerun the same “full regression suite” on different environments and think that is enough.

Instead, you need to think of what and why we test on different environments.


In Agile/Scrum, Dev/Test  teams “progressed” through these environments. In DevOps/Continuous Testing, you might be running your automation on many or all these environments concurrently.For example, what tests need to get run on the build environment? Why? What tests get run in the test environment? Why? What automated tests get run against, for example, VMs with various configurations, languages, browsers, mobile devices, etc.? What automated tests get run on the staging server? What is their purpose? What gets run on the staging server? What gets run in production?

Change your understanding of Regression

Regression as a term is probably the most misunderstood term in all software development. A Regression test is any test you run again. Some people in development think the regression suite or full regression is running every test run against a previous build or product. That is rarely the case. The regression suite is more likely rerunning every automated test. Well, not every test is automated, so only the automated tests are run again. Or regression could be happy path tests. Which probably, means only the main user tests POs must work for a customer to be happy. Not boundary or edge or rare or bug-finding cases- only the perfect, easy, no problem tests. How much confidence do you get from these?

Regression is often only priority 1 or 2 tests where lower priority tests are of less importance, not automated, not re-run, so therefore not all, Regression may only be the user story validation tests that were documented and automated along the way based on how much time the automation folks had to automate whatever they did rather than focus on long lasting, confidence building, bug finding regression suites. The fact is, regression means many things to different people.

This is particularly true where “automate everything” is a common mantra in continuous testing.  There is a speed to continuous delivery and continuous deployment that leaves no room for “back up plans”, the new description of regression has to be “lean and mean” fast and furious, (bug finding!), not bloated, or the same old  “whatever” we automated at the time.

For us to be testing more, and on more environments is pretty obvious. But the tests we are running have to change. Rerunning the same tests over and over on the same system with the same data against the same build does not help anyone.  We know we have a smoke test suite- a build validation suite that is run in the CI process on the dev or test environment.  We all have a “full regression” suite we normally run against the test environment. Here is where we need to begin reformulating.

For some teams, their automated “full regression” suite can take days to run.  Many “full regression” suites have gotten bloated and slow.

  • This is not OK. With the “Agile on steroids” mentality, automation thought/theory/practice needs to move from “bigger is better” to “lean and mean.” Continuous Testing suites need to be economical.
  • Teams will need to redesign, optimize and even cut the size of their automated suites for continuous testing rather than keep them get bloated and slow.


You will need various suites of tests. You will have a smoke test (however you define smoke) and a bigger set of tests, probably called regression, which are already part of your CI process.

Will you run either of these in production? I hope not. The goals of those suites are not what you want in production! An automated suite of tests you run in production must be extra careful to maintain the integrity of the production environment, and users while monitoring system behavior and availability.

 Will you run either of those suites against a big number of VMs? Do you need to? What tests will you run against each mobile device or emulator, for example? You need intelligent thought and a strategy for what test you need to run, how often, against which environments.

Understanding the purpose of each environment and the monitoring or information needed from that environment is key to selecting which tests get run. For example, you may have 4 “regression” suites- a fast smoke regression, a big full regression, a smaller “happy path”/main functionality, bug regression suite, main or core lean and mean regression suite, smaller  end-to-end transaction/integration style regression suite for the fully integration environments. A  smaller set of whatever you choose to run against production for monitoring. Each will be run at various points in the DevOps process on various environments. It’s important to remember, all this testing is in addition to the new feature development testing that happens inside your sprint. That testing continues as before.

All, meaningful, thoughtfully designed, prioritized tests- all regression tests- with different suite names.

Why do suites need to cut bloat and be lean and mean? For a few reasons.

  • Tests get old, and obsolete
  • Tests need maintenance.

Always be aware to cut poorly designed tests and failures. Tests that often fail or break easily need to be rethought and designed another way to find bugs

  • or breaks faster and be less fragile.
  • Immediate feedback.
  • An efficient set of tests will be run by other people on the team in continuous testing.
  • Get the biggest coverage with the smallest number of tests. That is, get the Biggest bang for the buck.

Testing in production

For all the years I have been in testing- it has always been a strict rule- “No testing in production” the last thing you want is to have the testing messing around on a live system/environment with customers and testing does something to muck up the system.


Only on rare occasions did the test team tread lightly onto live to do something that could only be done on live or unique to a live customer environment. This has partly been changed with the use of virtualization. This has entirely changed in continuous testing.

You may be asked to run a suite of tests to validate the migration to live, or monitor the live system. Whatever the case, you need to make sure you are testing what needs to be tested and no more on live. Make sure the whole team knows the risks of testing in the live production environment and the risk of not testing in the live environment,  and as we have talked about  your tests cannot interfere with the normal users workflow performance or operation of the system-be transparent.

A note here before we begin talking about  “automation.” Many people talk about automation in DevOps. Often, though, they do not mean test automation. They may be referring to task automation by the Ops team. Automating building environments, provisioning databases, populating user lists… these common tasks for many IT/Ops teams can now be scheduled, automated and  run significantly faster due to the flood of new tools focused on Ops and the “Infrastructure as Code” movement.

This is the flood of automation many people refer to.

Rethink automation

Continuous Testing relies on…continuous testing! Everything needs to be automated– or better, everything that can be automated, needs to be. This concept is the single key to successful continuous testing.

What tests need to be automated, what tests do not, how often they should be run and on which environments, how fast they are to run, the possibility the test will “fail”, the size of tests and their maintenance and other factors should impact what and how things will get automated. DevOps demands an entire re-thinking of your automation program. For example, it is common to  shrink your “full regression” to run faster and on more environments. Shrinking automation suites is an uncommon goal for many test teams.



The test strategy for continuous testing is a full reexamination of what gets tested where. This with the Lean Software development (LSD) idea of “quality at every step” is the foundation of a continuous test strategy. It involves reexamining what you have been calling regression and developing multiple lean and mean suites for immediate feedback.

It also means rethinking what and how you automate. Old style automation- even small isolated automated tests need to be reexamined and replaced by efficient, low level tests as well as longer customer-focused transaction workflows. Rethinking regression and automation is a big task for even the most effective teams. This is a big job.

Part 3 of this Continuous Testing in Dev Ops series deals with automation tool issues, environments and virtualization and cloud.

Checklist IDEAS for Automation in Continuous Testing and DevOps



Michael Hackett

Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006).
He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.


Leave a Reply

Your email address will not be published. Required fields are marked *