Achieving success with automated testing can be difficult. With the combination of Action Based Testing and TestArchitect, success is possible.
Avoiding something because it’s difficult will never get you anywhere in today’s day and age. Even though making automated testing successful in terms of both scalability and long-term maintainability is often regarded as a challenge, the Automated testing has become a must-have for Agile and CI/CD. With the arrival of highly automated build and release pipelines, the demand on Automation reliability has increased, because if the automated test breaks the pipeline breaks.
The good news is that technologies and processes have greatly improved. For example, there is much more room for cooperation in the agile approach; it has become more common to do a large part of testing at the API and unit test levels where Automation tends to be easier to create and maintain than testing via the user interface.
In particular, to make Automation successful in a CI/CD pipeline, our approach at LogiGear is a combination of a specific methodology, Action Based Testing (ABT), and a fitting toolset to support that methodology. ABT integrates the Automation focus into the test design, rather than just automating manual tests. ABT is a keyword-driven modular approach to automated testing, supported by our comprehensive product suite, TestArchitect™ (TA), that integrates test development, Automation, and test management in one package.
ABT and TA differs from many other Automation approaches available in that they place their main focus on test designs. Test designs play an important role as the driver to Automation success. The main goals for test design include:
- Simplicity
- Efficiency
- Manageability
- Maintainability
- Depth
- Coverage
Having a good test design not only makes tests more effective, it also makes them easier to understand, and maintain to fit them into CI/CD pipelines.
The image below shows an example of a test design that is not suitable for efficient and maintainable Automation. The test has many steps and verifies many outcomes but it is also hard to read and is sensitive to changes in the application under test. Even an experienced Automation Engineer will have a hard time automating tests like this. When you have tens of thousands of lines, or more, like the ones shown in the picture below, scalability becomes a problem.
Actions, Test Modules
In the Action Based Testing approach, tests are written as a sequence of keyword driven “actions” in a spreadsheet-like editor.
Each action has a name (action word) and any number of arguments. Actions are easy to work with and allows testers to quickly create a domain specific language for their tests. The actions can be implemented in code and with a sequence of other actions. In TestArchitect, we have a set of over 400 built-in actions as a starting point.
The purpose of a test defines what kind of actions to use. Most projects will have “business tests” and “interaction tests.” The business tests have a higher level of abstraction and the actions used are typically domain specific. Lower level steps are encapsulated in the actions. Interaction tests work with the details of dialogs or API’s and will use lower level actions that are not domain specific, often just simply the TA built-in actions.
Business tests focus on the functionality. In a business test, one should not expect navigational actions like “select menu item” or “click button.” By keeping interaction details like these navigational actions out of a test, these tests become more stable even if the system undergoes major maintenance. It also matters what aspect of a business is being tested. For example, a test module testing calculations for billing should not also test credit card processing.
An interaction test looks to see if a user, or another system, can interact with the application under test. In the example above, we check whether the OK button in a login dialog gets enabled when a username and password have been entered.
Interaction tests can be as varied as business tests. For example, one test might verify mandatory and optional fields while another might check if a dropdown box has the right values. When testing dropdown boxes, it’s common to not need to vary data much as long as the dropdown box has the right values and they are in the right order.
How to Use Action Based Testing in TestArchitect
Actions are easy to use and have an elegant expression power. Complex tasks, such as checking graphics or having multiple tests communicate with each other, are just a few tasks that can comfortably be expressed with actions. Such actions might not always be easy to implement and Automation engineering is an important task in the team. But once the technical challenge has been overcome, the resulting action can be re-used as many times as the tests demand.
In addition to using keyword-driven actions, ABT and TA take a modular approach to test design. Testers are encouraged to organize their tests into “test modules,” with each test module having a clearly defined and well-differentiated scope. The scope determines what actions to use and what checks to perform. Each module has a section for “test objectives,” or verbal statements that detail the scope of the module. Making test objectives is part science and part art. It’s possible to make and follow guidelines but you will have to analyze underlying requirements and specifications well and try to design test requirements that are both easy to access and easy to test. The rest of the test modules consist of test cases that address the objectives. Unlike what is common elsewhere, the core products in ABT are test modules, not test cases. In ABT, test cases are the outcome of the test creation process, not the input. The test cases in a test module form a narrative in which each test case leaves behind the pre-conditions for the next one.
A recommended way is to identify the test modules for a project early in the life-cycle. This can, for example, be done in a group session with Testers, Automation Engineers, Developers, product owners, and domain experts. The test modules thus forms a pre-structure that can be a guideline. In a Scrum project, the test modules will usually be developed and automated in the sprints. Sprint teams may make changes to the original test module line-up, which are not meant to be more than a guideline.
Let’s Take a Closer Look at How to Use ABT in TestArchitect
This is an example application of a music library. Clicking a song in the left-hand tree displays some information about that song on the right.
We can create a test module for the application in TestArchitect, in which we click a tree node, and then check some fields. The actions for example, check and click tree node, are built into the TestArchitect framework. See the image below for an example:
Looking closer at the test module above, however, notice that although the test is easily run, maintainability can become an issue. In particular, the tree path is hard-coded and could easily be different in a future version of the application. In the ABT methodology, details that do not fit in the scope of a test module are hidden.
By using a TA action definition, we can generalize this node path, making the test maintainable over time. Let’s create a new action that checks the bitrate field for a given song. The artist and song names are used to determine the node path, hiding it from the test that calls the action.
By doing this, the action now makes the focus of the test module functional, easy to read, maintainable, and easy to manage. The test lines abstract from their navigation and now this can be tied to the UI, but also to a service call, a database query, etc.
Sprints and Pipelines
A keyword-driven modular approach like ABT, supported by TA, can achieve two of the main challenges in Agile and DevOps:
1. Getting testing and Automation “done” at the end of a sprint so it can stay in sync with the rest of the team efforts
2. Planning and organizing the execution of tests in the pipeline
To get all testing done within a sprint takes commitment and cooperation in a team. A lot of ground can be covered by having test modules planned and using keywords as a practical means of communication and test expression. The test development will typically start with business tests, since they usually don’t need details of dialogs and navigation. Implementation of actions and the creation of interaction tests will come later, when UI and API developments are far enough along to know Automation details like internal names of screen elements and rough outlines of navigation. When available and within the same sprint, automated tests can also help drive development and even deepen the understanding of the functionality under test.
Depending on how many tests that need to be executed and in how many configurations, testing can become a bottleneck in a CI/CD pipeline, challenging the team with questions like what tests to run in each step of the pipeline and against which configurations. The modular approach of ABT can help make this determination, supported by a TA tool called “TAQueue” that can dispatch test modules for execution to available machines with a fitting configuration.
In addition, the slight differences in navigation and screen element identification that tend to come with different versions or configurations can be made transparent by a TA feature called “variations,” a feature applicable to actions and interface mappings.
To summarize, in order to achieve Test Automation success, both a good methodology and a proper tooling are necessary. A good focus on test planning, design, and close cooperation between project participants can lead to stable automated tests and an efficient, seamless execution of the tests in the pipeline.
Get TestArchitect Team Free
