Letter From The Editor – June 2020

Continuous Testing… what is it? When we first decided to do a magazine issue dedicated to the DevOps practice of Continuous Testing, I joked with someone: “It’s about testing continuously.”

And their reply was: “Yeah. What else would it be?”

I was joking, but clearly the joke didn’t land. Continuous Testing is about testing continuously, but it’s not that simple. The DevOps practice of Continuous Testing and what that looks like for your organization is going to be reliant on a few things, but let’s first talk about DevOps.

Remember, DevOps is, at its heart, about getting Development, Operations, and Testing all talking and collaborating early in the development process–namely, from the start of a project. In some cases, DevOps has morphed into choosing a toolset for Continuous Delivery. As important as that is, a toolset does not automatically mean you are doing “DevOps.” With more and earlier collaboration about testing and permanently breaking down the Silos there will be more testing, earlier in development.

Bringing the focus back to Continuous Testing, there are 2 cultural principles that have a greater influence on what Continuous Testing means: (1) shift-left, and (2) quality at every step. Shift-left is the development ideology that first gained attention in Agile by taking the testing stage, which was historically at the very end of the SDLC, and moving it earlier in the SDLC, thus shifting the testing “left” on the pipeline. This leads into quality at every step because by testing earlier (and more often) in the SDLC, you’re ensuring that after each phase, the quality of the software is good enough to be promoted to the next phase _ thus, quality at every step [of the SDLC].

First, you have to have a smooth, easy continuous integration/CI practice. This is build validation, meaning you get a build and run tests to see that you have a good build. First, you run the unit tests; then, run a smoke test consisting of higher-level tests focused more on integration than on individual units. Once the build is qualified, the team can go ahead and do more development and do more testing.

The thing about Continuous Testing, understanding both shift-left and quality at every step, is you do one thing and test it. Then, you do another thing and test that thing. The purpose here is if you break something but you only changed one thing, you know exactly why the break happened. This will make defect isolation and test it. Then, you do another thing and test that thing. The purpose here is if you break something but you only changed one thing, you know exactly why the break happened. This will make defect isolation and reworking pretty quick. In Continuous Testing, your testing is narrow because your change is narrow.

Think about it another way. If you are doing container architecture, one of the main benefits is that if you swap out one container with another container, you only have to test the container you changed out. You don’t have to run a 5-day automated GUI test suite consisting of 800,000 test cases across 5 platforms–you just run a test for the area that changed. You make a change and you test that change. Continuous Testing does not mean you run your full regression suite, again and again, every time you get a new build. You don’t need to test the whole thing.

You may ask, “Michael, are you saying if I change one part of a system, that there is no need to test another part of the system? Can’t one change to break a faraway piece?” To that, I say yes and no: It is true that one change can break another faraway piece; however, we all know that can happen. And that is a risk of speedy delivery and deployment.

But now this brings up the numbers game, and numbers are important.

Let’s say you have 10,000 automated tests… if it takes you only 20 minutes to run those 10,000 test cases, great! Go ahead and run your full regression suite every time you make any change or promote to the next environment if it’s only 20 minutes. But if those 10,000 tests take overnight to run, clearly you can’t run that test suite every time you push another build-out, or after every environment change. If your full regression suite consists of significant maintenance time, significant execution time, and significant cost, then you clearly cannot be continuously running that same full regression suite. It’s important to note that Continuous Testing is not about running tests continuously, but instead about delivering results continuously and quickly.

So, to refer back to my initial failed joke: Yes, Continuous Testing is, at a basic level, about testing continuously. However, there are clearly a lot of moving parts, requirements, and contingencies. There is no defined right or wrong way to Continuous Testing, but instead a bunch of grey areas that are dependent on your organization’s processes and goals–which makes it a really great topic for a 30+ page magazine!

The goal of this issue is to answer one question as thoroughly as we can: What is Continuous Testing? This issue brings together thought leadership from both inside and outside of LogiGear. The cover story, 5 Best Practices for Continuous Testing, was written by LogiGear CTO Hans Buwalda, in which he applies his decades of knowledge regarding test design and Automation to offer 5 tips for organizations who may be struggling with a CT implementation. Our infographic, The Beginner’s Guide to Continuous Testing, is a great resource for those who may be learning about or looking to implement CT for the first time. In the article Continuous Delivery: Everything You Need to Know, LogiGear Product Manager Thuc Nguyen covers “everything you need to know” (ha!) about Continuous Delivery, from definitions to tools choice, to testing in Continuous Delivery. This issue’s Blogger of the Month, Kristin Jackvony, shares her insight in Book Review: Continuous Testing for DevOps Professionals, which is a great guide written by industry experts for those seeking to optimize their CT program. Our strategic business partner Sauce Labs also contributed to this issue with Continuous Testing in the Retail Industry, which explores the growing “need for testing speed” in digital retail marketplaces. This issue also features the final installment of my 4-year Leader’s Pulse series; this final installment is essentially a culmination of my original intent for this series: What is the Value of a Manager’s Role in Modern Tech Organizations? And finally, to wrap it all up, our TestArchitect Team has created a step-by-step guide on How to Integrate The Jenkins CI Tool into TestArchitect.

We hope you enjoy this issue. Happy Testing!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

Methods and strategy have been my favorite topics since I started working in testing. It’s essentially engineering problem-solving. It’s both looking for efficiency and attempting to measure effectiveness. So, how do we develop a set of practices to solve our Software Testing engineering problems?
This is LogiGear magazine’s first issue on the big world of DevOps. DevOps is a very large topic. Just when you thought you were safe from more process improvement for a while—not so fast. There’s DevOps, Continuous Testing, Continuous Delivery and Continuous Deployment. In this issue, we are focusing on Continuous Testing, the part most ...
This is our first Trends issue in our 10- year history. Trends are important to help foresee what is on the horizon and coming next.
In our continuing effort to be the best source of information for keeping testers and test teams current, we have another issue to explore testing in Agile development. As Agile evolves, systemic problems arise and common rough situations become apparent. We want to provide solutions. For anyone who has worked on Agile projects, especially if ...
Our plan for the December LogiGear Magazine was to have a forward-looking Trends and Challenges issue. However, whilst assembling our September issue on SMAC, we realized the momentum SMAC was gaining in the industry. We had a large amount of content on our hands from a range of excellent contributors. Thus, we decided to split ...
There is a growing software development dynamic of teams without Testers. When I first went into Software Quality, I learned one thing right away: My role was user advocate. My main job was to find bugs. This is the Lean principle called Amplified Learning. We learn about behavior by testing. Even then, validation was not ...
In the November 2011 issue: Mobile Application Testing, I began my column with the statement, “Everything is mobile.” One year later the statement is even more true. More devices, more platforms, more diversity, more apps. It boggles the mind how fast the landscape changes. Blackberry has been kicked to the curb by cooler and slicker ...
On the whole, everyone wants to do a great job, have a better work environment, happy clients and customers, and to be employed by a company earning lots of money. All great goals! But this is not always the case. When it is not, you can suggest process improvements, better tool use, different estimating techniques, ...
What is testing in Agile? It’s analogous to three blind men attempting to describe an elephant by the way it feels to them. Agile is difficult to define and everyone has their own perspective of what Agile is. When it comes to testing and Agile the rules are what you make them. Agile is ideas ...
Software development projects are multifaceted. There is staffing and budget work. There are communication and team dynamics. There are project and process issues from what the customer wants, when they want it, revenue projections, and production dates. As part of my work in helping people deliver software, I get involved in all aspects mentioned above. ...
Testing Embedded systems and testing the Internet of Things could each have their own issue of LogiGear magazine. But these days they are referred to presupposing knowledge of the other, so we thought it would be a good idea to tackle the two together in this issue to give a broad understanding of the landscape ...
Testing tools – very important, very often overlooked, and very often where mistakes are made. First, the most common mistake people make about tools is thinking tools are only about test automation! False. Automation tools are merely one type testing tool. We will try to balance this issue between test automation tools and other test ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe