TOOLS
T1. What testing-support tools do you use? (Please check all that apply.)
| Response percent | Response count | |
| Bug tracking/issue tracking/defect tracking | 87.70% | 64 |
| Source control | 54.80% | 40 |
| Automation tool interface (to manage and run, not write automated tests) | 52.10% | 38 |
| Test case manager | 50.70% | 37 |
| Change request/change management/change control system | 47.90% | 35 |
| A full ALM (application lifecycle management) suite | 19.20% | 14 |
Result analysis: Thirteen percent do not use a bug tracking tool. This does not surprise me but it does many others that many test teams do not track their bugs!
About half of the respondents use a test case manager, and the same percentage uses a requirements manager or change control system. Half use a automation tool interface; these tools most commonly contain manual and automated test cases. Yet, only 20% use a full lifecycle ALM tool. A few years ago this number would have been much smaller.
With each passing year─especially as more teams go agile or offshore work─this number will dramatically increase.
T2. I would describe our bug (bug, issue, defects) tracking as:
| Response percent | Response count | |
| Effective | 37.70% | 26 |
| Very effective; has a positive impact on product quality | 34.80% | 24 |
| Adequate | 20.30% | 14 |
| A mess | 4.30% | 3 |
| Poor | 2.90% | 2 |
| Hurts product quality/has a negative impact on product quality | 0% | 0 |
T3. What type bug tracking tool do you use?
| Response percent | Response count | |
| Relational database tool (Bugzilla, TrackGear, TeamTrack, Team Test, ClearQuest, Homebuilt web-based or client server database) | 68.10% | 47 |
| ALM tool that includes defect tracking | 18.80% | 13 |
| Excel | 8.70% | 6 |
| 2.90% | 2 | |
| We do not track bugs | 1.40% | 1 |
Result analysis: This is a very positive move in our profession. Just a few years ago the number of teams using excel to track issues was significantly higher.
Excel is not an adequate issue tracking tool to sort, query, retrieve old issues from past releases, or control and manage access. With so many good and open source tools, there is no reason to be using a naive system.
T4. How many bug tracking systems do you use during a regular test project?
| Response percent | Response count | |
| 1 | 69.60% | 48 |
| Combination of tool and Excel and/or email | 14.50% | 10 |
| 2 | 11.60% | 8 |
| More than 2 | 4.30% | 3 |
Result analysis: The problem of multiple bug tracking tools is common. In this survey, almost 30% of teams use more than one bug tracking tool. Most problematic is that almost 15% who use a tool, use excel and email. I see this often in my consulting work. It always causes headaches.
One team will not use another team’s tool, developers have a work management tool and will not use the bug tracking tool so the test team has to use two tools, some remote team may not be allowed access to the internal tool so all their bugs get communicated in excel and email. It is a management problem, but it also lends to a more devious problem of giving the impression that testing is disorganized.
T5. How do you communicate and manage test cases?
| Response percent | Response count | |
| A relational database/repository focused on test case management (TCM, Silk Central, Rational Test Manager, TA, etc) | 41.50% | 27 |
| Excel | 21.50% | 14 |
| ALM tool that includes test case management | 20% | 13 |
| Word | 15.40% | 10 |
| We do not track, communicate or manage test cases | 1.50% | 1 |
Result analysis: The problem here is that almost 37% of teams are using MS-Word or Excel─that is dead data. It is difficult to share, edit, maintain, sort, query, and measure with these programs.
There are so many good test case management tools, some open source that make writing, editing/maintaining, sharing and measuring test cases so much easier. In my experience, there are very few good reasons not migrating to an easier and more sophisticated tool set.
There are also easy solutions to have test cases and bug tracking linked to the same tool. Test teams can graduate to a higher level of management, reporting and efficiency with tool sets.
T6. How are the test cases used? (Choose the MOST appropriate.)
| Response percent | Response count | |
| They are used only for testers to execute tests | 34.30% | 24 |
| They are used to measure and assess test coverage | 30% | 21 |
| They are used to assess proper test execution | 21.40% | 15 |
| They are used to measure and manage project progress | 14.30% | 10 |
| They are not used during the project | 0% | 0 |
Result analysis: For teams not using test cases for more than execution, it may be useful to know that it is very common for them to have other usage.
T7. If you use a test case management tool, how is it used?
| Response percent | Response count | |
| It is used to run our manual and automated tests | 57.40% | 31 |
| It is used only for manual tests | 40.70% | 22 |
| It is used to run only automated tests | 1.90% | 1 |
T8. If you have experience with success or failure regarding test tool use (ALM, bug tracking, test case management, automation tool interface, other) that is interesting or helpful to others, please write it below: (Comments from practitioners)
- “The best thing to do is manage the progress of the tests and see the bugs. You can measure the project’s health.”
- “I find Bugzilla reporting and commenting adequate communication most of the time. Its only problem is when immediate problems surface – at that point an email to appropriate parties telling them to look at Bugzilla usually works. So does walking over to the developer and showing them the issue.”
- “So far Jira was the best bug tracking tool.”
- “If you want people to use a TCM or bug management tool, make sure it has good performance and it’s simple.”
- “For a large project or program it is crucial to select a single method of tracking defects and what is considered defects versus ‘issues.’ This can lead to a great deal of confusion where defects identified as issues are not handled and addressed properly. I worked on a large project that various efforts had four different ways of tracking defects and issues. The result was that it was hard to assess the overall quality of the product that was being implemented.”
- “Testing should be driven by proven testing methodologies; not by the tool itself.”
- “Generating quality reports can be difficult using bug tracking systems.”
- Certain automation tools will not be suitable for certain type for projects.”
- “Test case management tools are not integrated to requirement management tools reason why our test cases are sometimes tested against obsolete functionality.”
- “Rally is very useful.”
- “Process discipline matters more than any tool.”
- “The tool is difficult to use for non-technical team members.”
