The Potential and Risks of Keyword Based Testing


Keyword Based Testing is gaining ground. More and more organizations see this model, in which tests are not scripted but written as a series of keywords with arguments, as a valuable alternative to record and playback, or scripting of tests. A good theoretical basis for keywords can be found in the well known Automation book Software Test Automation, by Dorothy Graham and Mark Fewster, and also in numerous articles and white papers on the LogiGear website. In this article I want to list some of the typical advantages and risks of keywords.

Advantages of Keyword Based Testing

With keyword methods, in particular when spreadsheets are used like with LogiGear’s Action Based Testing (ABT) method, tests can be developed by non-technical testers. And they will be able to produce more tests, and better tests. This way they can achieve more breadth in the tests, addressing more parts of the system in a shorter amount of time. They can also develop “deeper” test cases that test functionality in more detail, with more and more aggressive variations in the input.

The non-technical character of the test cases also allows business subject matter experts to get involved more easily, either in assessing the tests developed by testers or by creating good test cases themselves. Well-designed keyword based tests are less dependent on technical details of the system under test and are more stable over the long term. Contributions won’t have to be repeated. Once the knowledge of the experts is captured in the test cases, it remains a valuable asset for future projects.

Perhaps the most noticeable short term effect of keyword based testing is on the Automation side. Since the Automation focuses on the keywords, and not on the test cases built with those keywords, the implementation takes relatively little time and effort. This is even truer if a “multi-level” approach is used like we do in ABT. In that approach keywords are used as building blocks to create higher level keywords, and only the lowest level of keywords is responsible for interacting with the UI or other interfaces of the system under test. In a tool like TestArchitect, most of this lower level is already available, so we are seeing more and more projects being completely “script-less”.

A further positive effect on projects is that with keywords many tests can be completely developed early in the life cycle of a project, regardless of whether that project is following a traditional life cycle or an agile approach. This has tremendous advantages for the speed and manageability of the project. I wrote about this in my article Testing Under Pressure – Relieving the “Crunch Zone” in this newsletter.

Risks of Keyword Based Testing

The result of these advantages however is keywords are often brought in as a silver bullet: a solution for everything. Just roll in the tool and no attention is needed for testing anymore. This is not necessarily true! Like other powerful concepts, keywords can easily backfire. The method has pitfalls, pitfalls that need attention and take experience and understanding to avoid or address.

The most serious problems with keyword methods, and probably with other testing methods as well, arise from poor test design. In earlier newsletter articles I have described a number of principles that I call “The Three Holy Grails” of Test Design”. When these rules are not followed the result can be a voluminous, messy and hard to manage pile of tests that turn out to be hard to automate, thus loosing the main advantages of keyword methods. This situation can get so bad that keywords turn out worse than more traditional Automation techniques.

In practice keyword based projects can take longer than expected to develop and automate, usually because of poor test design. When there are changes in the system under test, the impact on the poorly designed tests can be greater than one would expect with keywords. When plans were made under the assumption that keywords are the “panacea”, reality can catch up and even completely derail a project schedule.

Another, less dramatic, effect of keyword based testing is also surprisingly common. The test cases developed with keywords are often shallow and boring. This is not unique to keywords, it is a broader effect of testing with pre-developed test cases, but somehow keywords seem to amplify it. Testers tend to follow a “mechanical” approach. For example, a set of system requirements is followed to the letter with exactly one test case per requirement. No attempt is made to break the system under test by creating uncommon situations and/or transactions. Even though there are many test design methods available, such as those outlined in the book Testing Computer Software, these are not used. Even more disappointing, testers don’t attempt to be creative in their work. This means the less obvious bugs are missed until the system is released.


My own conclusion, after many years of working with keywords, is that the method greatly exceeds any other in potential, but that it needs to be handled with care and attention. Reading the available materials, cooperating with others, and using creativity and imagination can let keywords lead you to great results.

Hans Buwalda Hans leads LogiGear’s research and development of test automation solutions, and the delivery of advanced test automation consulting and engineering services. He is a pioneer of the keyword approach for software testing organizations, and he assists clients in strategic implementation of the Action Based Testing™ method throughout their testing organizations. Hans is also the original architect of LogiGear’s TestArchitect™, the modular keyword-driven toolset for software test design, automation and management. Hans is an internationally recognized expert on test automation, test development and testing technology management. He is coauthor of Integrated Test Design and Automation (Addison Wesley, 2001), and speaks frequently at international testing conferences. Hans holds a Master of Science in Computer Science from Free University, Amsterdam.
Hans Buwalda
Hans Buwalda, CTO of LogiGear, is a pioneer of the Action Based and Soap Opera methodologies of testing and automation, and lead developer of TestArchitect, LogiGear’s keyword-based toolset for software test design, automation and management. He is co-author of Integrated Test Design and Automation, and a frequent speaker at test conferences.

The Related Post

Joe LuthyJoe Luthy has over 15 years product marketing experience in technology and Internet services for major companies including AT&T and Verisign. TestArchitect for Visual Studio is a keyword authoring platform extension designed specifically to enhance coded UI test automation in Visual Studio 2012. Microsoft’s Visual Studio’s ALM solution helps organizations manage the entire lifespan ...
Two powerful test methods for fast-paced development organizations As development teams have been pushed faster and into tighter scrum sprints, testing has burst through old development paradigms. Developers are being pressed to do more unit testing. Automated smoke tests are essential parts of CI (continuous integration) and full, automated regression suites are being run across ...
LogiGear Corporation   LogiGear Corporation provides global solutions for software testing, and offers public and corporate software-testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast and cost-effective results. Since 1994, LogiGear has worked with hundreds of companies from the ...
ABT and Keyword-driven testing – Similarities and Differences Both Keyword-driven testing and Action Based Testing (ABT) use a test authoring approach that separates much of the programming work of test automation from the actual test design. This allows tests to be developed earlier and aids in tests maintenance.
The best part of Action-Basted Testing is that it is for thinking people. It is intelligent and creative. It is a much saner way to evolve a testing project. All testers and quality engineers hear about Action-based testing (ABT) or keyword-driven testing somewhere. There are automation tools focused on keywords and actions. Maybe people have ...
Agile methods are becoming more and more popular and successful for developing IT systems. Typical properties of an Agile method, like Extreme Programming, involve continuous user involvement and emphasize on the testing role (‘Users’ may be the actual users of the system you are creating, customers, or business analysts who provide the requirements on behalf ...
To address the challenges and fears of implementing Automation in Agile projects, LogiGear CTO Hans Buwalda presents Action Based Testing as the answer.
Achieving success with automated testing can be difficult. With the combination of Action Based Testing and TestArchitect, success is possible. Avoiding something because it’s difficult will never get you anywhere in today’s day and age. Even though making automated testing successful in terms of both scalability and long-term maintainability is often regarded as a challenge, ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news