Introduction
Keyword Based Testing is gaining ground. More and more organizations see this model, in which tests are not scripted but written as a series of keywords with arguments, as a valuable alternative to record and playback, or scripting of tests. A good theoretical basis for keywords can be found in the well known Automation book Software Test Automation, by Dorothy Graham and Mark Fewster, and also in numerous articles and white papers on the LogiGear website. In this article I want to list some of the typical advantages and risks of keywords.
Advantages of Keyword Based Testing
With keyword methods, in particular when spreadsheets are used like with LogiGear’s Action Based Testing (ABT) method, tests can be developed by non-technical testers. And they will be able to produce more tests, and better tests. This way they can achieve more breadth in the tests, addressing more parts of the system in a shorter amount of time. They can also develop “deeper” test cases that test functionality in more detail, with more and more aggressive variations in the input.
The non-technical character of the test cases also allows business subject matter experts to get involved more easily, either in assessing the tests developed by testers or by creating good test cases themselves. Well-designed keyword based tests are less dependent on technical details of the system under test and are more stable over the long term. Contributions won’t have to be repeated. Once the knowledge of the experts is captured in the test cases, it remains a valuable asset for future projects.
Perhaps the most noticeable short term effect of keyword based testing is on the Automation side. Since the Automation focuses on the keywords, and not on the test cases built with those keywords, the implementation takes relatively little time and effort. This is even truer if a “multi-level” approach is used like we do in ABT. In that approach keywords are used as building blocks to create higher level keywords, and only the lowest level of keywords is responsible for interacting with the UI or other interfaces of the system under test. In a tool like TestArchitect, most of this lower level is already available, so we are seeing more and more projects being completely “script-less”.
A further positive effect on projects is that with keywords many tests can be completely developed early in the life cycle of a project, regardless of whether that project is following a traditional life cycle or an agile approach. This has tremendous advantages for the speed and manageability of the project. I wrote about this in my article Testing Under Pressure – Relieving the “Crunch Zone” in this newsletter.
Risks of Keyword Based Testing
The result of these advantages however is keywords are often brought in as a silver bullet: a solution for everything. Just roll in the tool and no attention is needed for testing anymore. This is not necessarily true! Like other powerful concepts, keywords can easily backfire. The method has pitfalls, pitfalls that need attention and take experience and understanding to avoid or address.
The most serious problems with keyword methods, and probably with other testing methods as well, arise from poor test design. In earlier newsletter articles I have described a number of principles that I call “The Three Holy Grails” of Test Design”. When these rules are not followed the result can be a voluminous, messy and hard to manage pile of tests that turn out to be hard to automate, thus loosing the main advantages of keyword methods. This situation can get so bad that keywords turn out worse than more traditional Automation techniques.
In practice keyword based projects can take longer than expected to develop and automate, usually because of poor test design. When there are changes in the system under test, the impact on the poorly designed tests can be greater than one would expect with keywords. When plans were made under the assumption that keywords are the “panacea”, reality can catch up and even completely derail a project schedule.
Another, less dramatic, effect of keyword based testing is also surprisingly common. The test cases developed with keywords are often shallow and boring. This is not unique to keywords, it is a broader effect of testing with pre-developed test cases, but somehow keywords seem to amplify it. Testers tend to follow a “mechanical” approach. For example, a set of system requirements is followed to the letter with exactly one test case per requirement. No attempt is made to break the system under test by creating uncommon situations and/or transactions. Even though there are many test design methods available, such as those outlined in the book Testing Computer Software, these are not used. Even more disappointing, testers don’t attempt to be creative in their work. This means the less obvious bugs are missed until the system is released.
Conclusion
My own conclusion, after many years of working with keywords, is that the method greatly exceeds any other in potential, but that it needs to be handled with care and attention. Reading the available materials, cooperating with others, and using creativity and imagination can let keywords lead you to great results.
| Hans Buwalda Hans leads LogiGear’s research and development of test automation solutions, and the delivery of advanced test automation consulting and engineering services. He is a pioneer of the keyword approach for software testing organizations, and he assists clients in strategic implementation of the Action Based Testing™ method throughout their testing organizations. Hans is also the original architect of LogiGear’s TestArchitect™, the modular keyword-driven toolset for software test design, automation and management. Hans is an internationally recognized expert on test automation, test development and testing technology management. He is coauthor of Integrated Test Design and Automation (Addison Wesley, 2001), and speaks frequently at international testing conferences. Hans holds a Master of Science in Computer Science from Free University, Amsterdam. |
