Letter From The Editor – June 2020

Continuous Testing… what is it? When we first decided to do a magazine issue dedicated to the DevOps practice of Continuous Testing, I joked with someone: “It’s about testing continuously.”

And their reply was: “Yeah. What else would it be?”

I was joking, but clearly the joke didn’t land. Continuous Testing is about testing continuously, but it’s not that simple. The DevOps practice of Continuous Testing and what that looks like for your organization is going to be reliant on a few things, but let’s first talk about DevOps.

Remember, DevOps is, at its heart, about getting Development, Operations, and Testing all talking and collaborating early in the development process–namely, from the start of a project. In some cases, DevOps has morphed into choosing a toolset for Continuous Delivery. As important as that is, a toolset does not automatically mean you are doing “DevOps.” With more and earlier collaboration about testing and permanently breaking down the Silos there will be more testing, earlier in development.

Bringing the focus back to Continuous Testing, there are 2 cultural principles that have a greater influence on what Continuous Testing means: (1) shift-left, and (2) quality at every step. Shift-left is the development ideology that first gained attention in Agile by taking the testing stage, which was historically at the very end of the SDLC, and moving it earlier in the SDLC, thus shifting the testing “left” on the pipeline. This leads into quality at every step because by testing earlier (and more often) in the SDLC, you’re ensuring that after each phase, the quality of the software is good enough to be promoted to the next phase _ thus, quality at every step [of the SDLC].

First, you have to have a smooth, easy continuous integration/CI practice. This is build validation, meaning you get a build and run tests to see that you have a good build. First, you run the unit tests; then, run a smoke test consisting of higher-level tests focused more on integration than on individual units. Once the build is qualified, the team can go ahead and do more development and do more testing.

The thing about Continuous Testing, understanding both shift-left and quality at every step, is you do one thing and test it. Then, you do another thing and test that thing. The purpose here is if you break something but you only changed one thing, you know exactly why the break happened. This will make defect isolation and test it. Then, you do another thing and test that thing. The purpose here is if you break something but you only changed one thing, you know exactly why the break happened. This will make defect isolation and reworking pretty quick. In Continuous Testing, your testing is narrow because your change is narrow.

Think about it another way. If you are doing container architecture, one of the main benefits is that if you swap out one container with another container, you only have to test the container you changed out. You don’t have to run a 5-day automated GUI test suite consisting of 800,000 test cases across 5 platforms–you just run a test for the area that changed. You make a change and you test that change. Continuous Testing does not mean you run your full regression suite, again and again, every time you get a new build. You don’t need to test the whole thing.

You may ask, “Michael, are you saying if I change one part of a system, that there is no need to test another part of the system? Can’t one change to break a faraway piece?” To that, I say yes and no: It is true that one change can break another faraway piece; however, we all know that can happen. And that is a risk of speedy delivery and deployment.

But now this brings up the numbers game, and numbers are important.

Let’s say you have 10,000 automated tests… if it takes you only 20 minutes to run those 10,000 test cases, great! Go ahead and run your full regression suite every time you make any change or promote to the next environment if it’s only 20 minutes. But if those 10,000 tests take overnight to run, clearly you can’t run that test suite every time you push another build-out, or after every environment change. If your full regression suite consists of significant maintenance time, significant execution time, and significant cost, then you clearly cannot be continuously running that same full regression suite. It’s important to note that Continuous Testing is not about running tests continuously, but instead about delivering results continuously and quickly.

So, to refer back to my initial failed joke: Yes, Continuous Testing is, at a basic level, about testing continuously. However, there are clearly a lot of moving parts, requirements, and contingencies. There is no defined right or wrong way to Continuous Testing, but instead a bunch of grey areas that are dependent on your organization’s processes and goals–which makes it a really great topic for a 30+ page magazine!

The goal of this issue is to answer one question as thoroughly as we can: What is Continuous Testing? This issue brings together thought leadership from both inside and outside of LogiGear. The cover story, 5 Best Practices for Continuous Testing, was written by LogiGear CTO Hans Buwalda, in which he applies his decades of knowledge regarding test design and Automation to offer 5 tips for organizations who may be struggling with a CT implementation. Our infographic, The Beginner’s Guide to Continuous Testing, is a great resource for those who may be learning about or looking to implement CT for the first time. In the article Continuous Delivery: Everything You Need to Know, LogiGear Product Manager Thuc Nguyen covers “everything you need to know” (ha!) about Continuous Delivery, from definitions to tools choice, to testing in Continuous Delivery. This issue’s Blogger of the Month, Kristin Jackvony, shares her insight in Book Review: Continuous Testing for DevOps Professionals, which is a great guide written by industry experts for those seeking to optimize their CT program. Our strategic business partner Sauce Labs also contributed to this issue with Continuous Testing in the Retail Industry, which explores the growing “need for testing speed” in digital retail marketplaces. This issue also features the final installment of my 4-year Leader’s Pulse series; this final installment is essentially a culmination of my original intent for this series: What is the Value of a Manager’s Role in Modern Tech Organizations? And finally, to wrap it all up, our TestArchitect Team has created a step-by-step guide on How to Integrate The Jenkins CI Tool into TestArchitect.

We hope you enjoy this issue. Happy Testing!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

There has been a tectonic shift in software development tools in just the past few years. Agile practices and increasingly distributed teams have been significant factors but, in my opinion, the main reason is a new and more intense focus on tools for testing driven by more complex software and shorter development cycles. There have ...
In every year since 2011, we have devoted one edition of our magazine to the topic of mobile testing. In this year’s issue on mobile, we focus on testing from the point of view of the user experience. Most teams start with UI testing, and it may seem basic — until you look at the ...
Testing the Software Car. As usual with the LogiGear Magazine, we are tackling a big subject. With our goal of having single-topic issues, we have the ability to grab and disseminate as much information as we can related to a current topic that is interesting and also on the frontier of Software Testing.   Some ...
Our plan for the December LogiGear Magazine was to have a forward-looking Trends and Challenges issue. However, whilst assembling our September issue on SMAC, we realized the momentum SMAC was gaining in the industry. We had a large amount of content on our hands from a range of excellent contributors. Thus, we decided to split ...
Every year, LogiGear Magazine devotes one full issue to Test Automation. We could do more than one, and perhaps even that would not be enough. The problems around automation have become increasingly complex. And now, automation is much more integrated into the software development process. For over a decade teams have been faced with “do ...
This is LogiGear magazine’s first issue on the big world of DevOps. DevOps is a very large topic. Just when you thought you were safe from more process improvement for a while—not so fast. There’s DevOps, Continuous Testing, Continuous Delivery and Continuous Deployment. In this issue, we are focusing on Continuous Testing, the part most ...
This is a very special issue of LogiGear Magazine. When we were putting together the Editorial Calendar for this year, we decided that instead of a technology issue, we would focus on the human side of quality and test engineering. We want to focus on individual Test Engineers and their jobs. We talked to a ...
I was just recently at a company that had a beautiful test architecture, framework, and Cucumber with tons of well-automated tests. But there was no good test management on top of the Cucumber tests, and they did not do a good job tagging the tests. Although almost everybody on the team could write and maintain ...
Happy New Year from LogiGear to those of us who celebrated New Years on January 1! And for our lunar calendar followers, an almost Happy New Year come February 3rd. We look forward to an exciting and full 2011 as its predecessor was a tough year for many in the software business. At LogiGear Magazine, ...
Integrated teams Something we’ve learned in the Covid-19 pandemic is that we have to work together-whatever together means. Very few teams stayed co-located; even teams in the same town worked at home. We’re all working remote. Hopefully all the thinking, tools, work and effort we put into having offshore teams work together benefited us here. ...
I have been excited about this issue since I included it in the 2011 editorial calendar. This issue of LogiGear Magazine dives into an exploration of agile automation—from the most efficient methods for test automation, to skill sets and better preparation for test teams, and even to understanding the variety of tools in question. We ...
There is a growing software development dynamic of teams without Testers. When I first went into Software Quality, I learned one thing right away: My role was user advocate. My main job was to find bugs. This is the Lean principle called Amplified Learning. We learn about behavior by testing. Even then, validation was not ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe