In order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.
Taxonomy is a systematic description of tool features and therefore requires proper understanding of those features. Chaotic taxonomy means you do not really grasp the technology, or usage of such tools.
Existing Taxonomies
Software testing lacks standards, and software test automation lacks them almost completely.
- The section on testing tools in software testing chapter of Wikipedia1 is very confusing–to say the least.
- ISO/IEC 29119 software testing standard2 is under development and far from complete.
- Software process standards such as TMMi3 or TPI4 state their tool taxonomy only indirectly–by stating vaguely what types of test tools are required for various maturity levels.
- Maturity Model for Automated Software Testing (MMAST)5 sounds promising, but is far from satisfactory, and almost totally unknown in software industry.
Last but not least, the most world-known de facto standard is ISTQB: International Software Testing Qualifications Board.6 Its syllabi do offer a relatively comprehensive test tools classification, but it is long from sufficient to be useful in practice.
The categories used there: test management, test execution, debugging and troubleshooting, fault seeding, simulation, static and dynamic analysis, keyword-driven test, performance testing and Web tools7 is more a mishmash of names belonging to different worlds (e.g. is keyword-driven tool something different than test execution tool?) rather than a useful taxonomy.
Tool Taxonomy for Agile Testing
Of course, a good and useful test tool taxonomy is the same for agile and for traditional development methods, but in agile methods you need it more, because product and project requirements change faster.
You cannot know all tools, but if you use a good taxonomy, than you have got a framework that enables you to locate and understand tools fast. Even a tool with a totally confusing name (that means, any tool—for some inexplicable reason both freeware and commercial test tools have names designed to impress and confuse, but never to help and guide the user), you are still able to understand roughly what it is in less than 30 seconds, provided you have the framework.
Using the framework presented below, you are no longer one-tool expert unable to see the context, nor you need be surprised by every new tool you encounter! But of course taxonomy does not replace detailed knowledge of given tools.
Outside the Framework
Some issues that are vital for understanding test tools do not fit nicely into tool taxonomy framework. We describe them shortly in this section.
- What test activities can be automated? The activities can range from intellectual to creative, and administrative to repeatable—you need to know which are best candidates for automation.
- Tools for testing products, projects and processes. Even if you seldom talk about project or process testing” (names like “audit” or “status monitoring” are used instead), such activities exist and are very important. In this article, we focus on product testing tools. But where does testing tools belong?
- Using tools is not yet automation. Tools must be used when human senses are not enough. Our eyes cannot see electrical or radio signals, and our hands cannot produce them. However, if you use a manually operated signal generator and a line listener, it is not yet test automation! This should be kept in mind.
- General types of test tools. Tools can be classified in a number of ways, but not all of them are useful in practice. However, you must be aware of these less important classifications, too, and not become confused.
- Dedicated test tools and generic tools used for testing. The border-line is often unclear, but you must not mix up things. For example, a word processor used for writing test case specifications, or a spreadsheet used for managing bug reports are very useful for automating some aspects of testing, but no one would call them “test tools.”
- Test tools can be software applications (most common usage), mechanical tools, electric or electronic measurement instruments and generators (this includes tools more commonly used for HW testing such as oscilloscopes or logic analyzers), cameras or other tracking instruments (used often for usability testing), chemicals sensors, and many more.
- Open source, freeware and commercial tools. Licensing schemes for commercial tools can be very different. Tool architecture and usage mode become more varied, too, as more and more cloud (software as a service, SAAS) tools appear.
- The distinction between emulators, simulators, test environment (even for manual testing), and debuggers, is not always easy to make, nor clear-cut.
- Tools with different degrees of intrusiveness from coverage measurement tools that instrument (change) the source code to electrical measurement instruments, whose only intrusiveness is probe effect in the form of resistance or inductance
- The level of tool integration. Increasingly, tools are not used alone but with many other tools as part of an integrated environment for automated testing. This creates many new issues, which are not covered in this article.
It must be kept in mind that real tools often span across many categories of these classifications. Some of the categories can be argued do not belong to testing at all, for example debugging or configuration management.
Action-based Framework
The most obvious and useful classification is dividing tools by what they do. It is comfortable to identify four categories here: test management, test execution, testware preparation and special test tools.
Test Management Tools
Test project management tools
- Tools for test reporting
- Change and configuration management tools
- Incident management tools
- Tools for build and integration
- Requirement tracking tools
Test Execution Tools
- Functionality 1: the ability to start dynamic test (send input data, activate signal, invoke a procedure, press mechanical sensor); tool’s “hands.”
- Functionality 2: the ability to record actual test result (output data, activated signal, radio signal); tools “eyes.”
- Functionality 3: the ability to compare actual and expected outcome; tool’s “brains.”
Typical test execution tools have all three functionalities, but there may well exist tools with only two of them or even one. Tools for dynamic analysis do not need any “hands,” and not tools for static testing and static analysis, either.
Testware Preparation Tools
Test data preparation tools: such tools can prepare input data, or pairs input— expected output data for dynamic testing, or can be used for populating databases, and for many other purposes.
- Test document preparation tools: there exist tools for producing test specifications from requirements or design (model-based testing). Then there is the gray zone of test reporting and incident management tools (that belong as well to the test management tools family), or tools for creating test logs (part of test execution tools).
- Test script preparation tools:
- Tools for script programming (with additional option for data-driven testing).
- Capture tools (typically, as part of capture-replay tools).
- Creating test scripts from source code (most commonly for unit testing); special cases here are automatic creating of test stubs and test drivers, and creating tests for markup languages (mostly HTML).
- Keyword-driven script generation: creating scripts from test specifications written in formal language.
- Special tools: tools for test coverage measurement and for fault injection.
Goal-based Tool Classification
What are the test goals?
Security Testing
To some extent, any test tool can be used for security testing, but there are some specialized tools as well. Some are used for penetration testing–dynamic design and execution of tests addressing some known security problems. But even static code analysis tools often contain a large number of rules for security issues.
Safety Testing
There are few tools used specially for testing system safety. Some of them are hardware tools.
Performance Testing
Performance (load, stress) testing tools are test execution tools with special capabilities for creating big load and measuring performance parameters exactly. They come in many different versions and for various platforms, and their sales make today the biggest part of test tool industry.
Conformance Testing
Various types of dynamic and static test execution tools, that focus on whether system behavior conforms to legal, standard or industry rules.
Tools for Testing Other Quality Attributes
Since at least 50-100 quality attributes can be identified (depending on the classifications), many more tool types can identified. This is not very useful, however.
Platform-based Test Tool Classification
There are thousands of different platforms for software systems, depending on hardware, operating system, API, network type, programming languages and other aspects. Most questions asked about software tools concern this issue, which can hide other, core aspects of test tools. But they are very important in practice. Some sub-classifications here: Web tools (very broad category), language-specific tools, tools for different operating systems (including the growing OS flora for mobile devices—Android, Symbian, iOS…), tools for various HW platforms, tools for testing embedded systems.
Level-based Tool Classification
Bottom-up classification framework: hardware tools, debuggers, tools for static code analysis, unit testing tools and frameworks, tools for system and acceptance testing (including usability).
- http://en.wikipedia.org/wiki/Software_testing#Testing_tools (all links checked on 12 March 2011)
- http://softwaretestingstandard.org/
- https://www.tmmi.org/
- See Test Process Improvement: A step-by-step guide to structured testing, Tim Koomen and Martin Pol
- http://sqa.fyicenter.com/art/A_Maturity_Model_for_Automated_Software_Testing.html
- http://istqb.org/display/ISTQB/Home
