Letter From The Editor – June 2020

Continuous Testing… what is it? When we first decided to do a magazine issue dedicated to the DevOps practice of Continuous Testing, I joked with someone: “It’s about testing continuously.”

And their reply was: “Yeah. What else would it be?”

I was joking, but clearly the joke didn’t land. Continuous Testing is about testing continuously, but it’s not that simple. The DevOps practice of Continuous Testing and what that looks like for your organization is going to be reliant on a few things, but let’s first talk about DevOps.

Remember, DevOps is, at its heart, about getting Development, Operations, and Testing all talking and collaborating early in the development process–namely, from the start of a project. In some cases, DevOps has morphed into choosing a toolset for Continuous Delivery. As important as that is, a toolset does not automatically mean you are doing “DevOps.” With more and earlier collaboration about testing and permanently breaking down the Silos there will be more testing, earlier in development.

Bringing the focus back to Continuous Testing, there are 2 cultural principles that have a greater influence on what Continuous Testing means: (1) shift-left, and (2) quality at every step. Shift-left is the development ideology that first gained attention in Agile by taking the testing stage, which was historically at the very end of the SDLC, and moving it earlier in the SDLC, thus shifting the testing “left” on the pipeline. This leads into quality at every step because by testing earlier (and more often) in the SDLC, you’re ensuring that after each phase, the quality of the software is good enough to be promoted to the next phase _ thus, quality at every step [of the SDLC].

First, you have to have a smooth, easy continuous integration/CI practice. This is build validation, meaning you get a build and run tests to see that you have a good build. First, you run the unit tests; then, run a smoke test consisting of higher-level tests focused more on integration than on individual units. Once the build is qualified, the team can go ahead and do more development and do more testing.

The thing about Continuous Testing, understanding both shift-left and quality at every step, is you do one thing and test it. Then, you do another thing and test that thing. The purpose here is if you break something but you only changed one thing, you know exactly why the break happened. This will make defect isolation and test it. Then, you do another thing and test that thing. The purpose here is if you break something but you only changed one thing, you know exactly why the break happened. This will make defect isolation and reworking pretty quick. In Continuous Testing, your testing is narrow because your change is narrow.

Think about it another way. If you are doing container architecture, one of the main benefits is that if you swap out one container with another container, you only have to test the container you changed out. You don’t have to run a 5-day automated GUI test suite consisting of 800,000 test cases across 5 platforms–you just run a test for the area that changed. You make a change and you test that change. Continuous Testing does not mean you run your full regression suite, again and again, every time you get a new build. You don’t need to test the whole thing.

You may ask, “Michael, are you saying if I change one part of a system, that there is no need to test another part of the system? Can’t one change to break a faraway piece?” To that, I say yes and no: It is true that one change can break another faraway piece; however, we all know that can happen. And that is a risk of speedy delivery and deployment.

But now this brings up the numbers game, and numbers are important.

Let’s say you have 10,000 automated tests… if it takes you only 20 minutes to run those 10,000 test cases, great! Go ahead and run your full regression suite every time you make any change or promote to the next environment if it’s only 20 minutes. But if those 10,000 tests take overnight to run, clearly you can’t run that test suite every time you push another build-out, or after every environment change. If your full regression suite consists of significant maintenance time, significant execution time, and significant cost, then you clearly cannot be continuously running that same full regression suite. It’s important to note that Continuous Testing is not about running tests continuously, but instead about delivering results continuously and quickly.

So, to refer back to my initial failed joke: Yes, Continuous Testing is, at a basic level, about testing continuously. However, there are clearly a lot of moving parts, requirements, and contingencies. There is no defined right or wrong way to Continuous Testing, but instead a bunch of grey areas that are dependent on your organization’s processes and goals–which makes it a really great topic for a 30+ page magazine!

The goal of this issue is to answer one question as thoroughly as we can: What is Continuous Testing? This issue brings together thought leadership from both inside and outside of LogiGear. The cover story, 5 Best Practices for Continuous Testing, was written by LogiGear CTO Hans Buwalda, in which he applies his decades of knowledge regarding test design and Automation to offer 5 tips for organizations who may be struggling with a CT implementation. Our infographic, The Beginner’s Guide to Continuous Testing, is a great resource for those who may be learning about or looking to implement CT for the first time. In the article Continuous Delivery: Everything You Need to Know, LogiGear Product Manager Thuc Nguyen covers “everything you need to know” (ha!) about Continuous Delivery, from definitions to tools choice, to testing in Continuous Delivery. This issue’s Blogger of the Month, Kristin Jackvony, shares her insight in Book Review: Continuous Testing for DevOps Professionals, which is a great guide written by industry experts for those seeking to optimize their CT program. Our strategic business partner Sauce Labs also contributed to this issue with Continuous Testing in the Retail Industry, which explores the growing “need for testing speed” in digital retail marketplaces. This issue also features the final installment of my 4-year Leader’s Pulse series; this final installment is essentially a culmination of my original intent for this series: What is the Value of a Manager’s Role in Modern Tech Organizations? And finally, to wrap it all up, our TestArchitect Team has created a step-by-step guide on How to Integrate The Jenkins CI Tool into TestArchitect.

We hope you enjoy this issue. Happy Testing!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

For everyone still celebrating holidays: Happy Lunar New Year! At this time of the year many teams and companies are starting new projects, new initiatives, and hiring new staff. LogiGear Magazine will continue to be the resource for you for better testing with much less stress! We are excited about the focus of this month’s ...
A lot has changed since I began staffing test projects. From hiring college students and interns for summer testing programs, to building networks of offshore teams around the world, and from having 24-hour work schedules to having instant crowdsourced public beta or bug bounty testing—things have changed.
Our plan for the December LogiGear Magazine was to have a forward-looking Trends and Challenges issue. However, whilst assembling our September issue on SMAC, we realized the momentum SMAC was gaining in the industry. We had a large amount of content on our hands from a range of excellent contributors. Thus, we decided to split ...
I once consulted for a company to give a week-long course on testing and QA. It was a survey course covering a wide range of topics. I was setting up and chatting with students in the room. One man came over to me and said: “I have been testing for 6 months and I am completely ...
Methods and strategy have been my favorite topics since I started working in testing. It’s essentially engineering problem-solving. It’s both looking for efficiency and attempting to measure effectiveness. So, how do we develop a set of practices to solve our Software Testing engineering problems?
Testers need to learn their craft and hone in on their skill set. That means building skills, sharpening their tools, and becoming creative detectives. There is no cookie-cutter tester and no best practice. The best circumstance is a fully-skilled, aggressive tester mixed with curiosity, nimbleness, and agility.
This is our first Trends issue in our 10- year history. Trends are important to help foresee what is on the horizon and coming next.
I have been excited about this issue since I included it in the 2011 editorial calendar. This issue of LogiGear Magazine dives into an exploration of agile automation—from the most efficient methods for test automation, to skill sets and better preparation for test teams, and even to understanding the variety of tools in question. We ...
Testing the Software Car. As usual with the LogiGear Magazine, we are tackling a big subject. With our goal of having single-topic issues, we have the ability to grab and disseminate as much information as we can related to a current topic that is interesting and also on the frontier of Software Testing.   Some ...
If you are reading this issue, you are probably aware of the impact on the business world of cloud computing. Most people do not have a good grasp on what the cloud is or how people and products can use it. BTW, you are already a cloud user. If your email is stored somewhere “on ...
Big and complex testing. What do these terms conjure up in your mind? When we added this topic to the editorial calendar, I had the notion that we might illustrate some large or complex systems and explore some of the test and quality challenges they present. We might have an article on: building and testing ...
We launched the first ever software testing conference in Vietnam, VISTACON. It was a resounding success, with well over 200 participants and 20+ speakers from around the globe; each speaking on a wide range of cutting-edge testing topics. In this month’s magazine, we have uploaded several video recordings of event presentations – giving our readers ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe