BrianA test plan should always include exploratory tests as part of the approach. Without them, the number of defects that find their way into production will always be higher.
Exploratory testing is a key testing technique that is often left out of formal test plan phases such as system testing, system integration, and regression. Instead, these phases favor planned, scripted tests that are easily repeatable and measurable.
While sometimes labelled as ‘ad-hoc’, and frowned upon in some circles because of the more unstructured nature, the truth is exploratory testing can be extremely fruitful in finding elusive bugs that may not otherwise have been discovered until user acceptance testing (UAT) or beyond.
There should therefore always be an allowance in the testing budget for conducting unplanned, exploratory tests – and the earlier the better.
Encourage Learning
According to the Wikipedia entry:
Exploratory testing … is concisely described as simultaneous learning, test design and test execution.
The key word is learning. One of the problems with scripted tests and script reuse is that the same functionality is covered, over and over again. This inhibits learning by narrowing the tester’s focus only to known areas, discouraging them from going off on their own and exploring other features or other ways of using the software.
People like to learn, and exploratory testing encourages that, making the process of testing more challenging and enjoyable, and leading to more productive testing.
“The real problem is not that the software hasn’t been tested, but that only key parts have been tested.”
Uncover Defects Earlier
A big problem with only running scripted tests is that they cover the same ground. While this may be fine for regression testing of core functionality, such script reuse does not fully test software changes in the way thorough exploratory testing can.
Relying only on scripted tests results in a larger than desired number of defects being missed during UAT.
As a tester, I have lost track of the number of times I have heard business users complain during UAT that “nobody has tested this”, when the real problem is not that the software hasn’t been tested, but that only key parts have been tested.
End-users are often very effective at finding bugs – in part this is probably due to their tendency to take a more exploratory approach as a result of their non-familiarity with the application.
Suggested Metrics for Exploratory Testing
Exploratory testing is sometimes left out of test plans because of uncertainty over what testing is being carried out, and therefore how to report progress. Many test managers focus on numerical progress reporting because of expectations from project and program managers who like to see a dashboard-style report that clearly shows the status of testing.
When there is no set number of planned exploratory tests to be carried out it can be difficult to show numerically whether or not you are on track, or whether you are achieving sufficient coverage of functionality.
One solution is to allocate a number of days for exploratory testing, and report a percentage complete based on the time elapsed. The level of coverage could be reported alongside that by completing a coverage matrix as tests are executed in defined areas of functionality.
Summary
However progress is reported, the key takeaway here is that a test plan should always include exploratory tests as part of the approach. Without them, the number of defects that find their way into UAT and production will always be higher.
By scheduling a number of days into the test plan for exploratory tests, more defects can be uncovered earlier in testing, sometimes dramatically improving the perceived quality of what gets handed over to the users.
