METHODS
M1. The test cases for your effort are based primarily on:
| Response percent | Response count | |
| Requirements documents | 61.3% | 46 |
| Discussions with users on expected use | 2.7% | 2 |
| Discussions with product, business analysts, and marketing representatives | 9.3% | 7 |
| Technical documents | 4% | 3 |
| Discussions with developers | 8% | 6 |
| My experience and subject or technical expertise | 12% | 9 |
| No pre-writing of test cases, I test against how the application is built once I see it | 2.7% | 2 |
| Guess work | 0% | 0 |
Result analysis: This confirms conventional wisdom. Over 60% of the respondents list requirements documents as the basis of their test cases. This brings up two important discussion points: 1) test cases can only be as good as or as thorough as the requirements and 2) past survey results exhibit that most testers are hired because of their subject matter expertise. This subject matter expertise is the primary basis for test case development for a far-distant 12%.
Some test teams complain regularly about requirements documents they receive. I would assume that reliance on subject matter expertise would have been a more common basis for test case development.
M2. Are you executing any application-level security and/or usability testing?
| Response percent | Response count | |
| Yes for both | 42.1% | 32 |
| Yes for usability | 23.7% | 18 |
| No for both | 23.7% | 18 |
| Yes for security | 10.5% | 8 |
M3. Are you conducting any performance and scalability testing?
| Response percent | Response count | |
| Yes | 74% | 54 |
| No | 26% | 19 |
M4. Does your group normally test for memory leaks?
| Response percent | Response count | |
| No | 52.1% | 38 |
| Yes | 47.9% | 35 |
M5. Does your group normally test for buffer overflows?
| Response percent | Response count | |
| No | 56% | 42 |
| Yes | 44% | 33 |
M6. Does your group normally test for multi-threading issues?
| Response percent | Response count | |
| No | 50.7% | 38 |
| Yes | 49.3% | 37 |
M7. Do you do any API testing?
| Response percent | Response count | |
| Yes | 62.7% | 47 |
| No | 37.3% | 28 |
Result analysis M2- M7: As test engineers, you need to know what types of testing need to be executed against your system even if you are not the person or team executing the tests. For example, performance tests are often executed by a different team separate from those that execute functional tests. However, your knowledge of test design and data design will help those executing these tests.
Knowledge in other tests can also help cut redundancy and shorten a test cycle. Most importantly, it becomes a serious problem if you are not executing these tests because you don’t know how or hope someone else will take over. If you are not doing memory tests because you think or hope the developers are, this is also a mistake.
API testing can find bugs earlier in the development cycle and has easier defect isolation. If API testing is not being practiced, you should have a good reason.
M8. What percentage of testing is exploratory/ad hoc and not documented?
| Response percent | Response count | |
| 10 – 33% (important part of our strategy) | 46.7% | 35 |
| Less than 10% (very little) | 26.7% | 20 |
| 33 – 66% (very important, approximately half, more than half) | 13.3% | 10 |
| 66% – 99% (our most important test method) | 10.7% | 8 |
| 0% (none) | 2.7% | 2 |
| 100% (all our testing is exploratory) | 0% | 0 |
Result analysis: With almost half responding that 10 -33% of exploratory testing plays an important part of their strategy, my biggest concern is that the project team knows and understands your use of exploratory testing and the difficulty in measuring it.
The high percentage calling it important and the difficulty measuring exploratory testing often leads to incorrectly sizing a test project or increasing risk in cutting schedule time allotted for exploratory testing.
I expected a bigger reliance on exploratory testing with only over a quarter responded, “less than 10 %.” Most team teams say they find most of their bugs running exploratory tests as opposed to validation test cases. This may still be true, but many test teams may lack the time to do exploratory tests.
M9. How effective is your exploratory/ad hoc testing?
| Response percent | Response count | |
| Somewhat effective, it is useful | 54.8% | 40 |
| Very effective, it is our best bug finding method | 39.7% | 29 |
| Not effective, it is a waste of time/money testing | 5.5% | 4 |
Result analysis: Almost 40% said their exploratory/ad hoc testing very effective and the best bug finding method. I agree and hope you communicate to your teams how effective it is and your reliance on it.
M10. How is exploratory/ad hoc testing viewed by project management?
| Response percent | Response count | |
| Somewhat effective, it is useful | 58.6% | 41 |
| Essential, necessary for a quality release | 20% | 14 |
| Very effective, it is our best bug finding method | 11.4% | 8 |
| Not effective, it is a waste of time/money testing | 10% | 7 |
Result analysis: Close to 60 % of respondents say management views the strategy as somewhat effective. In the previous questions, nearly the same percentage saw the testing as useful. This surprises me. Very often testers see ad hoc testing as more valuable than management who often see it as immeasurable and unmanageable.
M11. What is the goal of testing (test strategy) for your product?
| Response percent | Response count | |
| Validate requirements | 34.20% | 25 |
| Find bugs | 26% | 19 |
| Improve customer satisfaction | 23.30% | 17 |
| Cut customer support costs/help desk calls | 8.20% | 6 |
| Maximize code level coverage | 5.50% | 4 |
| Improve usability | 1.40% | 1 |
| Improve performance | 1.40% | 1 |
| Comply with regulations | 0% | 0 |
| Test component interoperability | 0% | 0 |
Result analysis: The number one answer for what is the goal of testing is validating requirements. This is a surprise. Typically, finding bugs is seen by testers as the goal and management, business analysts or marketing see validating requirements as the main goal and job of test teams.
Even with this, about half the respondents see finding bugs and improving customer satisfaction as the goal of testing. We see a few times in this survey a large reliance on requirements as the basis of testing. This can be a problem with anything less than great requirements!
M12. Which test methods and test design techniques does your group use in developing test cases? (You can choose more than one answer.)
| Response percent | Response count | |
| Requirements-based testing | 93.20% | 69 |
| Regression testing | 78.40% | 58 |
| Exploratory/ AdHoc testing | 68.90% | 51 |
| Scenario-based testing | 56.80% | 42 |
| Data driven testing | 40.50% | 30 |
| Equivalent class partitioning and boundary value analysis | 27% | 20 |
| Forced error testing | 25.70% | 19 |
| Keyword/Action-based testing | 17.60% | 13 |
| Path testing | 16.20% | 12 |
| Cause-effect graphing | 12.20% | 9 |
| Model-based testing | 10.80% | 8 |
| Attack-based testing | 9.50% | 7 |
M13. How do you do regression testing?
| Response percent | Response count | |
| Both | 47.30% | 35 |
| Manual | 33.80% | 25 |
| Automated | 14.90% | 11 |
Result analysis: For 33% of respondents, regression testing is purely manual. I see this commonly in development teams. There are so many good test automation tools on the market today that can be used on more platforms that teams not automating their regression tests ought to re-examine test automation. For all testers, test automation is a core job skill.
M14. Is your test environment maintained and controlled so that it contributes to your test effort?
| Response percent | Response count | |
| Yes | 83.1% | 49 |
| No | 16.9% | 10 |
M15. Are there testing or quality problems related to the environments with which you test?
| Response percent | Response count | |
| Yes | 64.4% | 47 |
| No | 35.6% | 26 |
M16. Are there testing or quality problems related to the data with which you test?
| Response percent | Response count | |
| Yes | 63% | 46 |
| No | 37% | 27 |
Result analysis M14- M16: Controlling test environments and test data is essential. Environments and data leading to testing problems is very common and very problematic! Building and maintaining great test environments and test data needs time and investment.
These answers confirm what I see in companies regularly─ problems in environments and data not getting resolved. With some time and perseverance, fixing these would greatly improve the effectiveness of testing and the trust of the test effort.
