Understanding Metrics in Software Testing

Introduction

Metrics are the means by which the software quality can be measured; they give you confidence in the product. You may consider these product management indicators, which can be either quantitative or qualitative. They are typically the providers of the visibility you need.

Metrics

Metrics usually fall into a few categories: project management (which includes process efficiency) and process improvement. People are often confused about what metrics they should be using. You may use different metrics for different purposes. For example, you may have a set of metrics that you use to evaluate the output of your test team. One such metric may be the project management measure of the number of bugs found. Others may be an efficiency measure of the number of test cases written, or the number of tests executed in a given period of time.

The goal is to choose metrics that will help you understand the state of your product.

Ultimately, when you consider the value of a metric, you need to ask if it provides visibility into the software product’s quality. Metrics are only useful if they help you to make sound business decisions in a timely manner. If the relevancy or integrity of a metric cannot be justified, don’t use it. Consider, for example, how management analysis and control makes use of financial reports such as profit/loss, cash flow, ratios, job costing, etc. These reports help you navigate your business in a timely manner. Engineering metrics are analogous, providing data to help perform analyses and control the development process. However, your engineers may not be the right people to give you the metrics you need to help in making business decisions, because they are not trained financial analysts. As an executive, you need to determine what metrics you want and tell your staff to provide them.

For example, coverage metrics are essential for your team. Coverage is the measure of some amount of testing. You could have requirements coverage metrics, platform coverage metrics, path coverage metrics, scenario coverage metrics, or even test plan coverage metrics, to name a few. Cem Kaner lists over 100 types of coverage measures in his paper “Negligence and Testing Coverage.” Before the project starts, it is important to come to agreement on how you will measure test coverage. Obviously, the more coverage of a certain type, the less risk associated with that type.

The goal is to choose metrics that will help you understand the state of your product. Wisely choose a handful of these metrics specific to your type of project and use them to give you visibility into how close the product is to release. The test group needs to be providing you plenty of useful information with these metrics.

Conclusion

The metrics provided by testing offer a major benefit to executives: visibility into the maturity and readiness for release or production, and visibility into the quality of the software product under development. This enables effective management of the software development process, by allowing clear measurement of the quality and completeness of the product.

Hung Nguyen Hung Nguyen co-founded LogiGear in 1994, and is responsible for the company’s strategic direction and executive business management. His passion and relentless focus on execution and results has been the driver for the company’s innovative approach to software testing, test automation, testing tool solutions and testing education programs. Hung is co-author of the top-selling book in the software testing field, “Testing Computer Software,” (Wiley, 2nd ed. 1993) and other publications including, “Testing Applications on the Web,” (Wiley, 1st ed. 2001, 2nd ed. 2003), and “Global Software Test Automation,” (HappyAbout Publishing, 2006). His experience prior to LogiGear includes leadership roles in software development, quality, product and business management at Spinnaker, PowerUp, Electronic Arts and Palm Computing. Hung holds a Bachelor of Science in Quality Assurance from Cogswell Polytechnical College, and completed a Stanford Graduate School of Business Executive Program.

Rob Pirozzi Over 20 years of sales, marketing, management, and technology experience in high technology with exposure to industries including financial services, healthcare, higher education, government, and manufacturing; demonstrating a strong track record of success. Proven ability to build and maintain strong relationships, contribute to target organization success, and deliver results. Website: http://www.robpirozzi.com/

Share:

Hung Q. Nguyen

Hung Nguyen co-founded LogiGear in 1994, and is responsible for the company’s strategic direction and executive business management. His passion and relentless focus on execution and results has been the driver for the company’s innovative approach to software testing, test automation, testing tool solutions and testing education programs. Hung is co-author of the top-selling book in the software testing field, “Testing Computer Software,” (Wiley, 2nd ed. 1993) and other publications including, “Testing Applications on the Web,” (Wiley, 1st ed. 2001, 2nd ed. 2003), and “Global Software Test Automation,” (HappyAbout Publishing, 2006). His experience prior to LogiGear includes leadership roles in software development, quality, product and business management at Spinnaker, PowerUp, Electronic Arts and Palm Computing. Hung holds a Bachelor of Science in Quality Assurance from Cogswell Polytechnical College, and completed a Stanford Graduate School of Business Executive Program.

Rob Pirozzi

Over 20 years of sales, marketing, management, and technology experience in high technology with exposure to industries including financial services, healthcare, higher education, government, and manufacturing; demonstrating a strong track record of success.