Letter from the Editor – March 2021

A while ago, I helped start a Software Quality Certificate Program as a part of the Software Engineering Program at the University of California, Santa Cruz Extension in Silicon Valley. I was on the Board of Advisors. While putting the curriculum together, a few people suggested a Measurement and Metrics course. Since I was teaching a few classes in the program, they asked me to write and lead the class. When they did, I did not realize how quickly––or how loudly––I replied “NO!”

The whole room froze and looked at me. I laughed. I told them that there was no way I would go near that class––not even with a 10-foot pole.

The problem with measurement is that every organization is unique, every product is at a different level of maturity and has its own meaningful measurement needs, team members have personal preferences to what informs their decisions, different tools have predesigned reports or dashboards of what they think you may want to know… what I’m getting at here is that it’s all context-driven. Reporting from QA and Test Teams serves many purposes, but reporting must also support the business goals. Thus, measurements and metrics must be relevant to these goals. The reporting needs to be actionable, meaning that the data informs decision-makers and arms them with what they need for:

  • Assessing product readiness (Go, no-go, not yet, etc.),
  • Ascertaining process efficiency (that testing and development are getting done as efficiently as needed), and
  • Accurately scoping project sizing, staffing, resourcing, tool needs, devices, etc.

Measurement and metrics reporting have a rocky history because measurements are easy to generate, so some people over-report, making it the reader’s job to see through and make sense of all of the data. Eliminating waste, being Lean, and cutting overhead is better management. “Just the Facts” and “Less is More” are better mantras for reporting for today.

Better than inventing some acronym, from my experience, the reporting you do must be:

  1. Correct. This may go without saying, but I could tell you horror stories of the incorrect or missing information or measurement leading to problems.
  2. Easy to generate or capture. Once you have defined the measurements to report to the team, they need to be easy to capture. If you regularly have to manually grab a piece of data from one tool and manually grab another piece of data from another tool, then put them into an Excel spreadsheet and calculate something to send the decision-makers or team members… think again. Capturing and calculating reporting needs to be automated.
  3. Understood. The team members that you are reporting this information to need to understand what it does and does not mean––and perhaps even what good numbers might look like and what bad numbers might look like. It is super common that measurements or metrics are reported and misjudged by various members of the team because they have never had them fully explained nor did they get an agreement on what is actually being measured and why.
  4. Used! The reporting is used for action. The right people do things, make decisions, make changes, get staff, get time, bless a release, or hold a release based on the numbers we are giving them. If it turns out that people are not making decisions based on these numbers…
  5. Reevaluate. Change them. If people are not taking action on what you report, if there is too much misunderstanding, or you are not getting the result you intended, get a different measure that’s easy to understand and that the team will be able to use for decision-making. Remember, there are no “best practices,” only continuous improvement.

Reporting is complex because it is always context-driven and often political. My biggest advice for reporting is to be Lean and be careful.

This issue is packed full of content aimed at helping QA leaders ensure their team’s efforts are actively (and accurately) communicated to business decision makers. Our cover story, How to Stop “Flying Blind” When Accounting for QA was written by LogiGear’s SVP of Sales Clayton Simmons and offers actionable insight for QA leaders who may be struggling to find the “perfect mix” of metrics when reporting to decision-makers. The Midpoint Crisis: How Automation Can Make More Manual Testing Work was written by Michael Larsen and explores how to prevent having to manually execute tasks as our Automation program matures. How to Be Seen was written by Kristin Jackvony and is aimed at entry-level QA professionals who are trying to find ways to make their work known and climb the corporate ladder. This issue’s Blogger of the Month, Wes Silverstein, explores the important role that QA has when it comes to planning Automation sprints in The Role of QA in Sprint Planning. Our infographic, 5 Incredibly Useful KPIs for Test Automation, explores 5 of our top-recommended testing metrics to use in your QA reports, including formulas and reasons for why they’re pertinent. Finally, check out the new features and functionality of TestArchitect Version 9.0 in TestArchitect Corner.

We hope you enjoy this issue––happy testing!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

This is a very special issue of LogiGear Magazine. When we were putting together the Editorial Calendar for this year, we decided that instead of a technology issue, we would focus on the human side of quality and test engineering. We want to focus on individual Test Engineers and their jobs. We talked to a ...
“Why do we need to understand a bunch of test methods? I write test cases from user stories or requirements, automate what I can and execute the rest manually, and its fine.” If this is your situation: good for you. If you are time crunched, if your automated tests have lost relevance, are hard to ...
Methods and strategy have been my favorite topics since I started working in testing. It’s essentially engineering problem-solving. It’s both looking for efficiency and attempting to measure effectiveness. So, how do we develop a set of practices to solve our Software Testing engineering problems?
A lot has changed since I began staffing test projects. From hiring college students and interns for summer testing programs, to building networks of offshore teams around the world, and from having 24-hour work schedules to having instant crowdsourced public beta or bug bounty testing—things have changed.
Testing the Software Car. As usual with the LogiGear Magazine, we are tackling a big subject. With our goal of having single-topic issues, we have the ability to grab and disseminate as much information as we can related to a current topic that is interesting and also on the frontier of Software Testing.   Some ...
In our continuing effort to be the best source of information for keeping testers and test teams current, we have another issue to explore testing in Agile development. As Agile evolves, systemic problems arise and common rough situations become apparent. We want to provide solutions. For anyone who has worked on Agile projects, especially if ...
This is our third issue concerning topics of Continuous Delivery (CD) and DevOps with the inclusion of Continuous Testing. DevOps has been around for a while and I hope the period of buzz is over and companies moving towards building a development pipeline have begun their process, including changing their test strategies.
As part of my work, I spend a lot of time at client’s sites and talk to various software development organizations. I am beginning to see a problem arise regarding Test Automation. There is too much automation! Surprised? While there are still many teams struggling to make progress with Test Automation, many teams have been doing ...
We launched the first ever software testing conference in Vietnam, VISTACON. It was a resounding success, with well over 200 participants and 20+ speakers from around the globe; each speaking on a wide range of cutting-edge testing topics. In this month’s magazine, we have uploaded several video recordings of event presentations – giving our readers ...
Testing tools – very important, very often overlooked, and very often where mistakes are made. First, the most common mistake people make about tools is thinking tools are only about test automation! False. Automation tools are merely one type testing tool. We will try to balance this issue between test automation tools and other test ...
As we settle into autumn, we’re taking the time to start some new traditions. This is LogiGear magazine’s first issue on SMAC. SMAC—social, mobile, analytics and cloud. We will be doing more issues in the next few years on these topics since so much of the product world is moving to this development stack.
Our plan for the December LogiGear Magazine was to have a forward-looking Trends and Challenges issue. However, whilst assembling our September issue on SMAC, we realized the momentum SMAC was gaining in the industry. We had a large amount of content on our hands from a range of excellent contributors. Thus, we decided to split ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe