Two powerful test methods for fast-paced development organizations
As development teams have been pushed faster and into tighter scrum sprints, testing has burst through old development paradigms. Developers are being pressed to do more unit testing. Automated smoke tests are essential parts of CI (continuous integration) and full, automated regression suites are being run across platforms, devices, virtual machines. With all the complexity, many teams have finally come to the conclusion that having a more methodological approach and strategy for test design, execution and maintenance is essential to keep up with the speed demands.
Testing is a dynamic business. Testing methods are developed and turned into acronyms that are tossed around as if everyone knows and understands them. After the initial hype most fizzle out. However, two methods in particular have proven valuable, and are popular and growing in use—Behavior Driven Development (BDD) and Action Based Testing (ABT).
The BDD and ABT methods
BDD is great. ABT is great. Both methods have been used extensively and undergone refinement over time. It’s important to understand that they are not interchangeable as test methods. One is not better than the other; the methods are quite different with different users, and different goals. To make effective use of either method requires fully understanding what each method is best suited to do and use it for that.
BDD was pioneered by Dan North in 2006 and grew out of Test Driven Development (TDD). ABT was pioneered in 2001 by Hans Buwalda and it grew out of keyword-driven testing. Each method is very good for what it is meant to do. Some situations are better suited for one than the other as they are practiced by different users for different purposes. This highlights a big problem in having software development as a sophisticated area of technical expertise: many people lack an understanding of various test methods with which to strategize a fully effective test project.
What is BDD?
Behavior driven development was developed out of a need for a guideline for effective and efficient TDD (test driven development), the eXtreme Programming practice. What is TDD? See TDD SideBar.
TDD is an idea, or paradigm. BDD is one way to execute it, and it’s a good way. It gives easy direction to programmers on how to do product specification and unit test with a low learning curve. Its purpose is to help the folks devising the system (i.e., the programmers) identify appropriate tests to write — that is, tests that reflect the behavior desired by the stakeholders. It involves the business and test teams through writing tests based on behaviors.
The main focus of BDD is to provide a clear understanding of software behavior by making use of natural language while writing test scripts.
My response [to TDD] is behavior-driven development (BDD). It has evolved out of established agile practices and is designed to make them more accessible and effective for teams new to agile software delivery. Over time, BDD has grown to encompass the wider picture of agile analysis and automated acceptance testing.
from Dan North “What’s in a Story?”
What is ABT?
ABT is a method based on keyword-driven testing that is focused on intelligent test design and organization that uses the concept of test modules to increase the efficiency of test development. Test modules provide a level of abstraction over test cases and make it possible to create well-defined test case flows. The top-down planning approach helps to create test cases that are free of unnecessary details and redundant checks. ABT also effectively separates the design of the tests from how they will be executed, allowing tests to be used for both manual and automated test execution.
While ABT has similarities, it is not the same as Keyword driven testing
Action Based Testing is an advanced keyword-driven test method that places emphasis on test design ahead of automation and execution. ABT encapsulates modular test design and action-driven methodology that makes large-scale, collaborative test development possible.
Hans Buwalda
In ABT, test cases are authored using readable keywords, or actions to symbolize functionality to be tested. It allows subject matter experts to create well defined tests and a single automation engineer to support a large group of testers by automating the execution.
Contrasting BDD and ABT
The two methods have a number of good aspects in common. Both ABT and BDD focus on business case driven behaviors for testing. They facilitate writing tests using good design practices without being concerned about how the tests will be executed—execution follows.
They are both written in a natural, domain specific, language that is not code and is easy for all people on the team to read and comprehend. Both methods:
- have some structure, but it’s not rigid
- have loose syntax
- are meant to be short and simple statements
- separate out test data
- focus on the separating test design from test execution
- can be used before any code is written, more Agile
- BDD and ABT allow automation to be done later.
One very important thing to understand about these methods is that they prescribe a top-down approach to test development. For most people this means there will be a steep learning curve. and for most projects more effort will need to be expended on up-front planning. For those who are used to equating code production with progress, this will seem counterproductive—at first. The end result will be that test case creation will be faster as they will be better thought-out and simpler. In ABT’s case, this also minimizes the effort to maintain tests.
Can you focus BDD and ABT only on Test Automation? Sure. It’s common. Some people think these are only methods for automation, but focusing only on test automation misses the full power of these methods. BDD and ABT are powerful methods to make sure you are building the right thing and have the ability to design tests early, at the beginning of a sprint, and leave the action automation to an automation engineer later after that functionality has been built.
Let’s look at constructing BDD and ABT tests to see the work products and what it looks like to see the full power of BDD and ABT and how different they are.
Writing BDD tests
One of the most initially appealing things about BDD is the simplicity of the documentation which is produced using a common natural language, most commonly Cucumber. http://en.wikipedia.org/wiki/Cucumber_(software)
A programmer reads a spec, requirement, or user story and then tries to describe the behavior before writing code. Dan North suggests starting with scenarios; how he defines scenarios is more commonly called a test objective.
BDD tests follow the format of a Scenario title for the test followed by a Given-When Then (GWT) syntax describing the product behavior. http://dannorth.net/whats-in-a-story/
Title (one line describing the story)
As a [role]
I want [feature]
So that [benefit]
Acceptance Criteria: (presented as Scenarios)
Scenario 1: Title
Given [context]
And [some more context]...
When [event]
Then [outcome]
And [another outcome]...
Scenario 2: ...
Below is an example of a BDD test scenario for cash withdraw from an ATM
Story: Account Holder withdraws cash
As an Account Holder
I want to withdraw cash from an ATM
So that I can get money when the bank is closed
Scenario 1: Account has sufficient funds
Given the account balance is \$100
And the card is valid
And the machine contains enough money
When the Account Holder requests \$20
Then the ATM should dispense \$20
And the account balance should be \$80
And the card should be returned
Scenario 2: Account has insufficient funds
Given the account balance is \$10
And the card is valid
And the machine contains enough money
When the Account Holder requests \$20
Then the ATM should not dispense any money
And the ATM should say there are insufficient funds
And the account balance should be \$20
And the card should be returned
Scenario 3: Card has been disabled
Given the card is disabled
When the Account Holder requests \$20
Then the ATM should retain the card
And the ATM should say the card has been retained
Scenario 4: The ATM has insufficient funds
...…
While this seems very easy, Gherkins can be, or can become, deceivingly complex.
When BDD is correctly used for TDD the test are straightforward and small since TDD focuses on small chunks of code-. However, the natural tendency is to have larger tests. Twelve to fifteen lines of “Given—When—and—and—and—Then—but—but” are more common than not.
When trying to adapt BDD for UI testing (not the primary intention of BDD), the tests can get long and complicated, which increases maintenance. The addition of “and—and— and—, but—but” adds complexity to tests when simplicity is the goal.
Breaking tests down into small, manageable (and maintainable) chunks is a discipline that has to be practiced to learn.
Writing ABT tests
In ABT, anyone on the project can write test objectives that identify what has to be tested, and then detail the task or workflow using actions (keywords). This style test can be created quickly regardless of how the tests will be executed. Also, with ABT, adding tests built upon already defined tests and actions is easy, scaling test development rapidly to high volume suites.
ABT starts with defining test modules which prescribe what tests will be included in each module. This is followed by defining test objectives in a cause and effect syntax. The test objective describes what each test case is supposed to do. From the objectives it’s a straightforward process to detail the test case for each object with actions and the appropriate data to execute the test.
Basic Syntax
Test Module
Test Objective: Cause and Effect
Test Case 1
Action data
Action data
Action data
...
Check data
Test Case 2
Action data
Action data
Action data
...
Check data
...and so on.
The example at the right shows a functional ABT test of cash withdraw from an ATM that follows a basic ABT template. The test scenario has three test cases that require only five actions. The ABT template can be found here.
When learning the syntax for writing ABT tests, the easiest part for everyone to grasp is the aspect of reusable actions to create low maintenance small tests. The part of ABT that is, for many people, the most effective is creating actions that can be reused and shared with teams. This is the keyword-driven aspect ABT was built on.
The ability to create and string together small, simple actions into bigger, feature-rich actions describing multi step tasks makes test case writing fast. Whole workflows or end-to-end scenarios can be contained in single actions this way, conceptually like using Lego blocks. This is also a very low maintenance approach since any change in the system under test only requires change or maintenance to one or just a few individual l actions. Any bigger action containing the smaller action would not be impacted.
More examples of Test Objective syntax: cause & effect format, short & descriptive.
– Refunded items should be returned to stock
– Replaced items should be returned to stock
– “clicking submit empties all fields”
– “if all fields are populated, ok is enabled”
– “ok becomes enabled if both first name and last name are specified”
– “in the case of a sports car: the screen specifies seconds to reach 60MPH”
– “in the case of a sports car: no lease price is available”
Best Uses
ABT and BDD are different. They solve different problems and have different best uses. Here are some examples.
Situation 1 – The scrum team does not fully understand the user stories or product. The user stories are not very detailed or the product has complicated domain specific tasks where the dev team may have only a high level knowledge of the workflow and domain. Use BDD to show behaviors of the system and get Product Owner buy-in. Big bonus for the team.
Situation 2 – Programmers do no unit tests, testers do all automation and regression. Programmers can write Given-When-Then statements and use a common unit test harness, such as JBehave for automated unit tests. Big bonus for the team.
Situation 3 – Test team lacks programming skill. Domain knowledgeable team (testers, POs, subject matter experts, business analysts) can write the various test objectives and conditions without concern for how the automation will get executed. An automation engineer using an action based automation tool, like TestArchitect, can automate and support about a dozen domain knowledge people. Big bonus for the team.
Situation 4 – A highly skilled, fully Agile team. Programmers can use BDD to better specify the system behavior. Action based test modules and test objectives can be developed at the same time. Once coding starts and you have product to be tested, the BDD will be turned into TDD/unit tests. The ABT tests (bigger, longer, more complex tests) can be automated. Big bonus for the team.
Situation 5 – Your product has multiple platforms or devices and you need a lot of high volume automation. Use ABT to design tests. Since ABT focuses on test design separate from the execution, the same tests can be executed across various platforms. An automation engineer, using a cross-platform tool, can change the implementation of the automation and the bulk of the work- developing the tests- might not change at all. Big bonus for the team.
Situation 6 – High maintenance cost for test automation. If you have a product that is very often in a state of change and/or constant redesign, your tests are probably high maintenance. Building a reusable, high-volume automated regression suite using a test method focused on low maintenance and high reusability is a must. ABT which focuses on small action level maintenance is best for this. Big bonus for the team.
Summary
Test Methods are useful sets of practices for test design, communication, measuring coverage, execution, documentation, modeling, and various other aspects of the development process.
Better understanding BDD (primarily used by programmers) and ABT, (primarily used by test engineers) will help you choose the best test method for various situations. ABT & BDD are not interchangeable. While BDD is great to specify a product and build unit tests, ABT is great at build high volume, cross platform, robust automation suites. Each method is very good at what it is meant to do. Used correctly both methods will result in better testing.
Get TestArchitect Team for Free
