Letter from the Editor – March 2021

A while ago, I helped start a Software Quality Certificate Program as a part of the Software Engineering Program at the University of California, Santa Cruz Extension in Silicon Valley. I was on the Board of Advisors. While putting the curriculum together, a few people suggested a Measurement and Metrics course. Since I was teaching a few classes in the program, they asked me to write and lead the class. When they did, I did not realize how quickly––or how loudly––I replied “NO!”

The whole room froze and looked at me. I laughed. I told them that there was no way I would go near that class––not even with a 10-foot pole.

The problem with measurement is that every organization is unique, every product is at a different level of maturity and has its own meaningful measurement needs, team members have personal preferences to what informs their decisions, different tools have predesigned reports or dashboards of what they think you may want to know… what I’m getting at here is that it’s all context-driven. Reporting from QA and Test Teams serves many purposes, but reporting must also support the business goals. Thus, measurements and metrics must be relevant to these goals. The reporting needs to be actionable, meaning that the data informs decision-makers and arms them with what they need for:

  • Assessing product readiness (Go, no-go, not yet, etc.),
  • Ascertaining process efficiency (that testing and development are getting done as efficiently as needed), and
  • Accurately scoping project sizing, staffing, resourcing, tool needs, devices, etc.

Measurement and metrics reporting have a rocky history because measurements are easy to generate, so some people over-report, making it the reader’s job to see through and make sense of all of the data. Eliminating waste, being Lean, and cutting overhead is better management. “Just the Facts” and “Less is More” are better mantras for reporting for today.

Better than inventing some acronym, from my experience, the reporting you do must be:

  1. Correct. This may go without saying, but I could tell you horror stories of the incorrect or missing information or measurement leading to problems.
  2. Easy to generate or capture. Once you have defined the measurements to report to the team, they need to be easy to capture. If you regularly have to manually grab a piece of data from one tool and manually grab another piece of data from another tool, then put them into an Excel spreadsheet and calculate something to send the decision-makers or team members… think again. Capturing and calculating reporting needs to be automated.
  3. Understood. The team members that you are reporting this information to need to understand what it does and does not mean––and perhaps even what good numbers might look like and what bad numbers might look like. It is super common that measurements or metrics are reported and misjudged by various members of the team because they have never had them fully explained nor did they get an agreement on what is actually being measured and why.
  4. Used! The reporting is used for action. The right people do things, make decisions, make changes, get staff, get time, bless a release, or hold a release based on the numbers we are giving them. If it turns out that people are not making decisions based on these numbers…
  5. Reevaluate. Change them. If people are not taking action on what you report, if there is too much misunderstanding, or you are not getting the result you intended, get a different measure that’s easy to understand and that the team will be able to use for decision-making. Remember, there are no “best practices,” only continuous improvement.

Reporting is complex because it is always context-driven and often political. My biggest advice for reporting is to be Lean and be careful.

This issue is packed full of content aimed at helping QA leaders ensure their team’s efforts are actively (and accurately) communicated to business decision makers. Our cover story, How to Stop “Flying Blind” When Accounting for QA was written by LogiGear’s SVP of Sales Clayton Simmons and offers actionable insight for QA leaders who may be struggling to find the “perfect mix” of metrics when reporting to decision-makers. The Midpoint Crisis: How Automation Can Make More Manual Testing Work was written by Michael Larsen and explores how to prevent having to manually execute tasks as our Automation program matures. How to Be Seen was written by Kristin Jackvony and is aimed at entry-level QA professionals who are trying to find ways to make their work known and climb the corporate ladder. This issue’s Blogger of the Month, Wes Silverstein, explores the important role that QA has when it comes to planning Automation sprints in The Role of QA in Sprint Planning. Our infographic, 5 Incredibly Useful KPIs for Test Automation, explores 5 of our top-recommended testing metrics to use in your QA reports, including formulas and reasons for why they’re pertinent. Finally, check out the new features and functionality of TestArchitect Version 9.0 in TestArchitect Corner.

We hope you enjoy this issue––happy testing!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

For everyone still celebrating holidays: Happy Lunar New Year! At this time of the year many teams and companies are starting new projects, new initiatives, and hiring new staff. LogiGear Magazine will continue to be the resource for you for better testing with much less stress! We are excited about the focus of this month’s ...
Big and complex testing. What do these terms conjure up in your mind? When we added this topic to the editorial calendar, I had the notion that we might illustrate some large or complex systems and explore some of the test and quality challenges they present. We might have an article on: building and testing ...
Automation is a mantra in testing. Anyone associated with software development wants more test automation, but it’s often misunderstood. People who do test automation know how difficult it can be. But some people do not understand that automation is code, and that it needs to have architecture and design just like production code. They do ...
This is our third issue concerning topics of Continuous Delivery (CD) and DevOps with the inclusion of Continuous Testing. DevOps has been around for a while and I hope the period of buzz is over and companies moving towards building a development pipeline have begun their process, including changing their test strategies.
Software development projects are multifaceted. There is staffing and budget work. There are communication and team dynamics. There are project and process issues from what the customer wants, when they want it, revenue projections, and production dates. As part of my work in helping people deliver software, I get involved in all aspects mentioned above. ...
Everything is mobile. What else can we say? Everything. If your product or service is currently not, it will be very soon. As Apple says: “There’s an app for that.” There is an app for everything. The race for mobile apps has consumed the software development world. I did a few projects at Palm Computing in the ...
Test automation is a big topic. There are so many different areas to talk about: tool choice, jumpstart, cross platform, services, cloud… Each of these areas have changed so much in the recent past that they could each be worth their own magazine issue.
How do you test software? How do you validate it? How do you find bugs? These are all good questions anyone on your project team or anyone responsible for customers may ask you. Can you articulate your test strategy─not your test process, but explain your approach to testing? I find that this can be a ...
There is a growing software development dynamic of teams without Testers. When I first went into Software Quality, I learned one thing right away: My role was user advocate. My main job was to find bugs. This is the Lean principle called Amplified Learning. We learn about behavior by testing. Even then, validation was not ...
As fast as Mobile is growing, the platform is still immature and is evolving at a very rapid pace. While there are whole countries that have migrated large government services to mobile, countries ranging from Estonia to Turkey to Kenya have many longtime mobile users have yet to use mPay or other mobile payment systems. ...
Methods and strategy have been my favorite topics since I started working in testing. It’s essentially engineering problem-solving. It’s both looking for efficiency and attempting to measure effectiveness. So, how do we develop a set of practices to solve our Software Testing engineering problems?
I spend about half my work time in the role of a consultant assessing, auditing and examining software development team practices and processes for the purpose of process improvement. I am regularly surprised to find teams that lack basic skills, management support, tools, information, access to users, Product Owners and to developers. And yet they’re ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe