Letter from the Editor – March 2021

A while ago, I helped start a Software Quality Certificate Program as a part of the Software Engineering Program at the University of California, Santa Cruz Extension in Silicon Valley. I was on the Board of Advisors. While putting the curriculum together, a few people suggested a Measurement and Metrics course. Since I was teaching a few classes in the program, they asked me to write and lead the class. When they did, I did not realize how quickly––or how loudly––I replied “NO!”

The whole room froze and looked at me. I laughed. I told them that there was no way I would go near that class––not even with a 10-foot pole.

The problem with measurement is that every organization is unique, every product is at a different level of maturity and has its own meaningful measurement needs, team members have personal preferences to what informs their decisions, different tools have predesigned reports or dashboards of what they think you may want to know… what I’m getting at here is that it’s all context-driven. Reporting from QA and Test Teams serves many purposes, but reporting must also support the business goals. Thus, measurements and metrics must be relevant to these goals. The reporting needs to be actionable, meaning that the data informs decision-makers and arms them with what they need for:

  • Assessing product readiness (Go, no-go, not yet, etc.),
  • Ascertaining process efficiency (that testing and development are getting done as efficiently as needed), and
  • Accurately scoping project sizing, staffing, resourcing, tool needs, devices, etc.

Measurement and metrics reporting have a rocky history because measurements are easy to generate, so some people over-report, making it the reader’s job to see through and make sense of all of the data. Eliminating waste, being Lean, and cutting overhead is better management. “Just the Facts” and “Less is More” are better mantras for reporting for today.

Better than inventing some acronym, from my experience, the reporting you do must be:

  1. Correct. This may go without saying, but I could tell you horror stories of the incorrect or missing information or measurement leading to problems.
  2. Easy to generate or capture. Once you have defined the measurements to report to the team, they need to be easy to capture. If you regularly have to manually grab a piece of data from one tool and manually grab another piece of data from another tool, then put them into an Excel spreadsheet and calculate something to send the decision-makers or team members… think again. Capturing and calculating reporting needs to be automated.
  3. Understood. The team members that you are reporting this information to need to understand what it does and does not mean––and perhaps even what good numbers might look like and what bad numbers might look like. It is super common that measurements or metrics are reported and misjudged by various members of the team because they have never had them fully explained nor did they get an agreement on what is actually being measured and why.
  4. Used! The reporting is used for action. The right people do things, make decisions, make changes, get staff, get time, bless a release, or hold a release based on the numbers we are giving them. If it turns out that people are not making decisions based on these numbers…
  5. Reevaluate. Change them. If people are not taking action on what you report, if there is too much misunderstanding, or you are not getting the result you intended, get a different measure that’s easy to understand and that the team will be able to use for decision-making. Remember, there are no “best practices,” only continuous improvement.

Reporting is complex because it is always context-driven and often political. My biggest advice for reporting is to be Lean and be careful.

This issue is packed full of content aimed at helping QA leaders ensure their team’s efforts are actively (and accurately) communicated to business decision makers. Our cover story, How to Stop “Flying Blind” When Accounting for QA was written by LogiGear’s SVP of Sales Clayton Simmons and offers actionable insight for QA leaders who may be struggling to find the “perfect mix” of metrics when reporting to decision-makers. The Midpoint Crisis: How Automation Can Make More Manual Testing Work was written by Michael Larsen and explores how to prevent having to manually execute tasks as our Automation program matures. How to Be Seen was written by Kristin Jackvony and is aimed at entry-level QA professionals who are trying to find ways to make their work known and climb the corporate ladder. This issue’s Blogger of the Month, Wes Silverstein, explores the important role that QA has when it comes to planning Automation sprints in The Role of QA in Sprint Planning. Our infographic, 5 Incredibly Useful KPIs for Test Automation, explores 5 of our top-recommended testing metrics to use in your QA reports, including formulas and reasons for why they’re pertinent. Finally, check out the new features and functionality of TestArchitect Version 9.0 in TestArchitect Corner.

We hope you enjoy this issue––happy testing!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

I led the Editor’s Note in our very first mobile issue with “Everything is mobile”, but it is now way beyond what we thought. Mobile has come to mean only the smart phone, mobility is the word that describes everything a smart phone enables you to do. Mobility is more than a device! Mobility is ...
We launched the first ever software testing conference in Vietnam, VISTACON. It was a resounding success, with well over 200 participants and 20+ speakers from around the globe; each speaking on a wide range of cutting-edge testing topics. In this month’s magazine, we have uploaded several video recordings of event presentations – giving our readers ...
In our continuing effort to be the best source of information for keeping testers and test teams current, we have another issue to explore testing in Agile development. As Agile evolves, systemic problems arise and common rough situations become apparent. We want to provide solutions. For anyone who has worked on Agile projects, especially if ...
A lot has changed since I began staffing test projects. From hiring college students and interns for summer testing programs, to building networks of offshore teams around the world, and from having 24-hour work schedules to having instant crowdsourced public beta or bug bounty testing—things have changed.
Automation is a mantra in testing. Anyone associated with software development wants more test automation, but it’s often misunderstood. People who do test automation know how difficult it can be. But some people do not understand that automation is code, and that it needs to have architecture and design just like production code. They do ...
I was just recently at a company that had a beautiful test architecture, framework, and Cucumber with tons of well-automated tests. But there was no good test management on top of the Cucumber tests, and they did not do a good job tagging the tests. Although almost everybody on the team could write and maintain ...
I spend about half my work time in the role of a consultant assessing, auditing and examining software development team practices and processes for the purpose of process improvement. I am regularly surprised to find teams that lack basic skills, management support, tools, information, access to users, Product Owners and to developers. And yet they’re ...
Testing the Software Car. As usual with the LogiGear Magazine, we are tackling a big subject. With our goal of having single-topic issues, we have the ability to grab and disseminate as much information as we can related to a current topic that is interesting and also on the frontier of Software Testing.   Some ...
Hello everyone – I’m hoping each one of us is having a great October. This time of the year is always my favorite, with the changing of the seasons, Fall was always my favorite time of year; it signified change and renewal – but I don’t want to digress to much from what’s going on ...
There is a growing software development dynamic of teams without Testers. When I first went into Software Quality, I learned one thing right away: My role was user advocate. My main job was to find bugs. This is the Lean principle called Amplified Learning. We learn about behavior by testing. Even then, validation was not ...
As we settle into autumn, we’re taking the time to start some new traditions. This is LogiGear magazine’s first issue on SMAC. SMAC—social, mobile, analytics and cloud. We will be doing more issues in the next few years on these topics since so much of the product world is moving to this development stack.
I once consulted for a company to give a week-long course on testing and QA. It was a survey course covering a wide range of topics. I was setting up and chatting with students in the room. One man came over to me and said: “I have been testing for 6 months and I am completely ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe