METRICS AND MEASUREMENTS
MM1. Do you have a metric or measurement dashboard built to report to your project team?
| Response percent | Response count | |
| Yes | 69% | 49 |
| No | 31% | 22 |
Result analysis: Anything worth doing is worth measuring. Why would almost 1/3 of teams not measure? Is the work not important or respected? Does the team not care about the work you do?
I am not measurement obsessed but when test groups do not report measurements back to the project team, it is very often the sign of a bigger problem.
MM2. If yes, are these metrics and measurements used or ignored?
| Response percent | Response count | |
| This information along with schedule decides product release | 43.40% | 23 |
| Some attention is paid to them | 26.40% | 14 |
| This information decides product release | 18.90% | 10 |
| Minimal attention is paid to them | 5.70% | 3 |
| They are ignored | 5.70% | 3 |
Result analysis: It is good to see that over 60% of teams reporting metrics use the information to decide release. Test team metrics should help decide product release.
As with the previous question, if the project team is not paying attention to test measurements, that is often the sign of a problem.
MM3. What testing specific metrics do you currently collect and communicate? (You may select multiple answers.)
| Response percent | Response count | |
| Test case progress | 84.10% | 53 |
| Bug/defect/issue metrics (total # open, # closed, # high priority, etc.) | 82.50% | 52 |
| Requirements coverage | 49.20% | 31 |
| Defect density | 34.90% | 22 |
| Defect aging | 25.40% | 16 |
| Root cause analysis | 25.40% | 16 |
| Code coverage | 22.20% | 14 |
| Defect removal rates | 22.20% | 14 |
| Requirements stability/requirements churn | 17.50% | 11 |
| Hours tested per build | 11.10% | 7 |
MM4. What methods/metrics do you use to evaluate the project status? (Comments from respondents.)
- “Too many.”
- “Bug Rate Trends and Test Case Coverage (Feature and Full Regression)”
- “Test case progress, defect status.”
- “Earned & Burned”
- “Number and severity of remaining open bugs. ’Finger in the air’ (tester intuition that some areas need more testing).”
- “Executed test cases.”
- “Defect counts and severity along with test case completion.”
- “Bug/defect/issue metrics (total # open, # closed, # high priority, etc.)”
- “Test case progress, defect density, and requirement coverage.”
- “Requirement coverage and test completed.”
- “Bugs found vs. fixed.”
- “Test coverage and defect metrics.”
- “Defects open along with test case execution progress.”
- “None.”
- “Are all the features ‘tested.'”
- “Track change proposals and outcomes (P/F) for all changes by project, by application, by developer, and by week.”
- “Exit criteria are agreed upon upfront, then metrics report progress against those. “
- “Test case complete %, pass/fail %, # of high-severity bugs.”
- “Schedule deviation, defects detected at each stage of project.”
MM5. Do you collect any metrics to evaluate the focus of the test effort, that is, to audit if you are running the right tests?
| Response percent | Response count | |
| Yes | 56.9% | 37 |
| No | 43.1% | 28 |
Result analysis: It is a step higher in responsibility and ownership when test teams evaluate their own work for effectiveness. Good work!
MM6. Do the metrics you use most help you:
| Response percent | Response count | |
| Release better product | 31.30% | 20 |
| Improve the development and test process | 26.60% | 17 |
| Do more effective/efficient testing | 23.40% | 15 |
| They do not help | 18.80% | 12 |
Result analysis: With 80% respondents using metrics to improve is great! More teams can be using metrics to point out problems, improve risk reporting, and give greater visibility into testing. Also, if your team is not capturing any measurements for the difficult reasons, it is safe to say your work is not respected, no one cares, or the team is purely schedule driven regardless of what testing finds. I recommend you start a metrics program to improve your job skills!
MM7. Do you measure regression test effectiveness?
| Response percent | Response count | |
| Yes | 59.1% | 39 |
| No | 40.9% | 27 |
Result analysis: Regression test effectiveness is a growing issue in testing. Numerous teams have been doing large scale test automation for many years now. Regression suites can become large, complex or difficult to execute. Many of the regression tests may be old, out of date, or are no longer effective. Yet, teams are often afraid to reduce the amount of regression tests.
At the same time, running very large regression test suites can take up too much bandwidth and impede a project. If you are having problems with your regression tests, start investigating their effectiveness.
MM8. How much of an effort is made to trace requirements directly to your test cases and defects?
| Response percent | Response count | |
| The test team enters requirements into a tool and traces test cases and bugs to them | 41.50% | 27 |
| We write test cases and hope they cover requirements; there is no easy, effective way to measure requirements coverage | 18.50% | 12 |
| The product/marketing team enters all requirements into a tool. Test cases and bugs are traced back to the requirements. Coverage is regularly reported | 15.40% | 10 |
| We do not attempt to measure test coverage against anything | 15.40% | 10 |
| We trace test cases in a methodical, measurable, effective way against code (components, modules or functions) | 9.20% | 6 |
Result analysis: Tracing requirements to a test case does not guarantee a good product. Measuring requirements coverage has become an obsession of some teams.
The big issue for teams doing this is that the test case can only be as good as the requirements. Gaps in the requirements, incomplete or even bad requirements will help assure a problem product. Tracing test cases to problematic requirements will do no one any good.
This practice can be really useful for measuring requirements churn, bug density, or root cause analysis. Tracing or mapping requirements can be a good method of assessing the relevance of your tests.
MM9. If code coverage tools are used on your product, what do they measure?
| Response percent | Response count | |
| Code-level coverage is lines of code, statements, branches, methods, etc. | 55.3% | 21 |
| Effectiveness of test cases (test cases mapped to chunks/blocks/lines of code) | 44.7% | 17 |
MM10. How do you measure and report coverage to the team
| Response percent | Response count | |
| Test plan/test case coverage | 45.30% | 29 |
| Requirements coverage | 25% | 16 |
| We do not measure test coverage | 15.60% | 10 |
| Code coverage | 7.80% | 5 |
| Platform/environment coverage | 6.30% | 4 |
| Data coverage | 0% | 0 |
Result analysis: Coverage, however you define this is crucial to report to the team. It is the communication of where we are testing. This is the crux of the discussion of what is enough testing.
