Letter From The Editor – June 2020

Continuous Testing… what is it? When we first decided to do a magazine issue dedicated to the DevOps practice of Continuous Testing, I joked with someone: “It’s about testing continuously.”

And their reply was: “Yeah. What else would it be?”

I was joking, but clearly the joke didn’t land. Continuous Testing is about testing continuously, but it’s not that simple. The DevOps practice of Continuous Testing and what that looks like for your organization is going to be reliant on a few things, but let’s first talk about DevOps.

Remember, DevOps is, at its heart, about getting Development, Operations, and Testing all talking and collaborating early in the development process–namely, from the start of a project. In some cases, DevOps has morphed into choosing a toolset for Continuous Delivery. As important as that is, a toolset does not automatically mean you are doing “DevOps.” With more and earlier collaboration about testing and permanently breaking down the Silos there will be more testing, earlier in development.

Bringing the focus back to Continuous Testing, there are 2 cultural principles that have a greater influence on what Continuous Testing means: (1) shift-left, and (2) quality at every step. Shift-left is the development ideology that first gained attention in Agile by taking the testing stage, which was historically at the very end of the SDLC, and moving it earlier in the SDLC, thus shifting the testing “left” on the pipeline. This leads into quality at every step because by testing earlier (and more often) in the SDLC, you’re ensuring that after each phase, the quality of the software is good enough to be promoted to the next phase _ thus, quality at every step [of the SDLC].

First, you have to have a smooth, easy continuous integration/CI practice. This is build validation, meaning you get a build and run tests to see that you have a good build. First, you run the unit tests; then, run a smoke test consisting of higher-level tests focused more on integration than on individual units. Once the build is qualified, the team can go ahead and do more development and do more testing.

The thing about Continuous Testing, understanding both shift-left and quality at every step, is you do one thing and test it. Then, you do another thing and test that thing. The purpose here is if you break something but you only changed one thing, you know exactly why the break happened. This will make defect isolation and test it. Then, you do another thing and test that thing. The purpose here is if you break something but you only changed one thing, you know exactly why the break happened. This will make defect isolation and reworking pretty quick. In Continuous Testing, your testing is narrow because your change is narrow.

Think about it another way. If you are doing container architecture, one of the main benefits is that if you swap out one container with another container, you only have to test the container you changed out. You don’t have to run a 5-day automated GUI test suite consisting of 800,000 test cases across 5 platforms–you just run a test for the area that changed. You make a change and you test that change. Continuous Testing does not mean you run your full regression suite, again and again, every time you get a new build. You don’t need to test the whole thing.

You may ask, “Michael, are you saying if I change one part of a system, that there is no need to test another part of the system? Can’t one change to break a faraway piece?” To that, I say yes and no: It is true that one change can break another faraway piece; however, we all know that can happen. And that is a risk of speedy delivery and deployment.

But now this brings up the numbers game, and numbers are important.

Let’s say you have 10,000 automated tests… if it takes you only 20 minutes to run those 10,000 test cases, great! Go ahead and run your full regression suite every time you make any change or promote to the next environment if it’s only 20 minutes. But if those 10,000 tests take overnight to run, clearly you can’t run that test suite every time you push another build-out, or after every environment change. If your full regression suite consists of significant maintenance time, significant execution time, and significant cost, then you clearly cannot be continuously running that same full regression suite. It’s important to note that Continuous Testing is not about running tests continuously, but instead about delivering results continuously and quickly.

So, to refer back to my initial failed joke: Yes, Continuous Testing is, at a basic level, about testing continuously. However, there are clearly a lot of moving parts, requirements, and contingencies. There is no defined right or wrong way to Continuous Testing, but instead a bunch of grey areas that are dependent on your organization’s processes and goals–which makes it a really great topic for a 30+ page magazine!

The goal of this issue is to answer one question as thoroughly as we can: What is Continuous Testing? This issue brings together thought leadership from both inside and outside of LogiGear. The cover story, 5 Best Practices for Continuous Testing, was written by LogiGear CTO Hans Buwalda, in which he applies his decades of knowledge regarding test design and Automation to offer 5 tips for organizations who may be struggling with a CT implementation. Our infographic, The Beginner’s Guide to Continuous Testing, is a great resource for those who may be learning about or looking to implement CT for the first time. In the article Continuous Delivery: Everything You Need to Know, LogiGear Product Manager Thuc Nguyen covers “everything you need to know” (ha!) about Continuous Delivery, from definitions to tools choice, to testing in Continuous Delivery. This issue’s Blogger of the Month, Kristin Jackvony, shares her insight in Book Review: Continuous Testing for DevOps Professionals, which is a great guide written by industry experts for those seeking to optimize their CT program. Our strategic business partner Sauce Labs also contributed to this issue with Continuous Testing in the Retail Industry, which explores the growing “need for testing speed” in digital retail marketplaces. This issue also features the final installment of my 4-year Leader’s Pulse series; this final installment is essentially a culmination of my original intent for this series: What is the Value of a Manager’s Role in Modern Tech Organizations? And finally, to wrap it all up, our TestArchitect Team has created a step-by-step guide on How to Integrate The Jenkins CI Tool into TestArchitect.

We hope you enjoy this issue. Happy Testing!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

There has been a tectonic shift in software development tools in just the past few years. Agile practices and increasingly distributed teams have been significant factors but, in my opinion, the main reason is a new and more intense focus on tools for testing driven by more complex software and shorter development cycles. There have ...
In our continuing effort to be the best source of information for keeping testers and test teams current, we have another issue to explore testing in Agile development. As Agile evolves, systemic problems arise and common rough situations become apparent. We want to provide solutions. For anyone who has worked on Agile projects, especially if ...
On the whole, everyone wants to do a great job, have a better work environment, happy clients and customers, and to be employed by a company earning lots of money. All great goals! But this is not always the case. When it is not, you can suggest process improvements, better tool use, different estimating techniques, ...
Happy New Year from LogiGear to those of us who celebrated New Years on January 1! And for our lunar calendar followers, an almost Happy New Year come February 3rd. We look forward to an exciting and full 2011 as its predecessor was a tough year for many in the software business. At LogiGear Magazine, ...
Methods and strategy have been my favorite topics since I started working in testing. It’s essentially engineering problem-solving. It’s both looking for efficiency and attempting to measure effectiveness. So, how do we develop a set of practices to solve our Software Testing engineering problems?
Every organization goes through times when the internal, or home team, cannot execute the testing project easily or quickly enough. The reasons are many, from the lack of an effective test strategy to low automation engineering skill, to staff positions going unfilled due to a great job market. With everyone working and very few people ...
How do you test software? How do you validate it? How do you find bugs? These are all good questions anyone on your project team or anyone responsible for customers may ask you. Can you articulate your test strategy─not your test process, but explain your approach to testing? I find that this can be a ...
The Greek philosopher Heraclitus of Ephesus (c. 500 BCE) is credited with saying, “The only constant is change.”   This is a statement that, more than 2,000 years later, still holds true. Today, we are in a time of great change. Everything is in flux. The fact is, we are always in a state of change even if ...
“Why do we need to understand a bunch of test methods? I write test cases from user stories or requirements, automate what I can and execute the rest manually, and its fine.” If this is your situation: good for you. If you are time crunched, if your automated tests have lost relevance, are hard to ...
API testing– an old school technology gets way cool again. APIs and testing them is nothing new; the technology has been around for decades. The most basic definition of an API is an exposed function— a producer (person or company) writes a function and exposes it so that others, consumers, can use it. We copy ...
I have been excited about this issue since I included it in the 2011 editorial calendar. This issue of LogiGear Magazine dives into an exploration of agile automation—from the most efficient methods for test automation, to skill sets and better preparation for test teams, and even to understanding the variety of tools in question. We ...
Our plan for the December LogiGear Magazine was to have a forward-looking Trends and Challenges issue. However, whilst assembling our September issue on SMAC, we realized the momentum SMAC was gaining in the industry. We had a large amount of content on our hands from a range of excellent contributors. Thus, we decided to split ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe