Automation and Requirements

Introduction

In many of the Test Automation projects that we are involved with using our Action-Based Testing methodology, management has expressed a need to relate tests and test results to system requirements. The underlying thought is that automation will create extra possibilities to control the level of compliance to requirements of the system under test. In this article I will explore some of the aspects and considerations regarding tests and requirements.

System and Test Requirements with Action-Based Testing

Action-Based Testing encourages testers to create “test requirements” as part of the test modules (spreadsheets with the tests to be carried out). The test requirements govern the scope of test cases that are in the test module; in other words, they detail the intention the tester has with the cases described.

For example, a system requirement is typically formulated like “the system should.” A test requirement typically reads as “test if.” As a short hand, we often leave the “test if” out, so a test requirement could read “the system does not accept password shorter than 6 characters,” to be understood as “test if the system does not accept password shorter than 6 characters.”

Now the next question is: how do the test requirements relate to system requirements? In many cases the link is very direct. As illustrated in the example above on minimum password length, the test requirement will relate one-on-one to a system requirement. In other cases a system requirement is a composite statement, which would lead to several test requirements.

For example “the system should enforce that passwords are at least 6 characters in length and contain at least 1 digit,” would lead to (1) “the system does not accept passwords shorter than 6 characters,” and (2) “the system requires a password to have at least 1 digit.”

Good Test Design: Going Beyond System Requirements

Another possibility is that a test requirement specifies that a test be created that does not directly relate to any system requirement. For example, a valid test requirement could state “test with 5 users from different states who are using the same mailbox.” This means the test should create a test case in which the described situation is simulated, but there is very likely no system requirement talking about 5 users.

Test requirements that are unrelated to system requirements illustrate an important consideration regarding system requirements and tests: A test designer should use system requirements as input and make sure all of them are covered in test cases, but a good test designer should go beyond that narrow mission. System requirements mostly focus on single elements of a system under test. A tester should make sure that the elements of the system work well together, and that the system can handle complex situations that the developer might not necessarily have thought about, because they were not specifically addressed in system requirements.

One potential benefit of Automation is to provide information related to system requirements. However, this needs to be carefully organized to be effective. Consider the following picture in which some relationships have been illustrated.

The solid arrows show how system requirements are, directly or indirectly, related to test requirements, which in turn are related to test cases, which in turn are related to test results. A test tool like TestArchitect will give you the possibility to establish relations 1-3 in the above picture as follows:

  1. In the details panel for a test requirement it is possible to register a “source”, this field can contain the identifier of the system requirement to which the test requirement relates.
  2. The same detail panel also shows the test cases in the test modules (after they have been created and the module has been checked in at least once), and checkmarks to indicate which one(s) cover the test requirement.
  3. The relation from the test results to the test case will be shown in the test results. Once the test result is checked in the test case details panel will show the latest result.

To easily obtain the consolidated relation (4) make sure the “source” fields of the test requirements are consistently populated, the test requirements are related to test cases and test run results (of successful runs) are checked in. This will make it easy to use the report generator in TestArchitect to show the test results per system requirement: define a report that shows test requirements, and as grouping field specify the “source” field.

Conclusion

This article has outlined that good test design has 3 major components:

  • Creating test cases that directly relate to system requirements
  • Creating test cases that indirectly relate to system requirements
  • Creating test cases that do not directly relate to system requirements

The third item is often overlooked and can be very critical in a successful Test Automation effort.

More Information

  • More information about Action-Based Testing may be found here.
  • More information on TestArchitect may be found here.
Hans Buwalda
Hans Buwalda, CTO of LogiGear, is a pioneer of the Action Based and Soap Opera methodologies of testing and automation, and lead developer of TestArchitect, LogiGear’s keyword-based toolset for software test design, automation and management. He is co-author of Integrated Test Design and Automation, and a frequent speaker at test conferences.

The Related Post

LogiGear Magazine – October 2010
An Overview of Four Methods for Systematic Test Design Strategy Many people test, but few people use the well-known black-box and white-box test design techniques. The technique most used, however, seems to be testing randomly chosen valid values, followed by error guessing, exploratory testing and the like. Could it be that the more systematic test ...
LogiGear Magazine – April 2013 – Test Automation
The huge range of mobile devices used to browse the web now means testing a mobile website before delivery is critical.
Many organizations rely on HP Quality Center to design test plans and track test results. TestArchitect’s Quality Center integration makes working with QC as easy as pie. TestArchitect (TA) is a three-in-one tool for Test Management, Test Development, and Test Automation. Users can create and manage test assets, execute tests, track and analyze test results, ...
From automotive Software Testing standards, testing techniques, and process, this article is an in-depth guide for those looking to transfer their existing skills to this exciting industry. For the Software Car, autonomous driving gets most of the hype, but most overlook the fact that there is so much more to Software Testing for the automotive ...
This article was developed from concepts in the book Global Software Test Automation: A Discussion of Software Testing for Executives, by Hung Q. Nguyen, Michael Hacket and Brent K. Whitlock Introduction The top 5 pitfalls encountered by managers employing software Test Automation are: Uncertainty and lack of control Poor scalability and maintainability Low Test Automation ...
Based in Alberta, Canada, Jonathan Kohl takes time out of his busy schedule to discuss his views on software testing and automation.
The growing complexity of the Human-Machine Interface (HMI) in cars offers traditional testers an opportunity to capitalize on their strengths. The human-machine interface (HMI) is nothing new. Any user interface including a graphical user interface (GUI) falls under the category of human-machine interface. HMI is more commonly being used to mean a view into the ...
All too often, software development organizations look at automating software testing as a means of executing existing test cases faster. Very frequently there is no strategic or methodological underpinning to such an effort. The approach is one of running test cases faster is better, which will help to deliver software faster. Even in organizations that ...
This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives. Introduction There are many potential pitfalls to Manual Software Testing, including: Manual Testing is slow and costly. Manual tests do not scale well. Manual Testing is not consistent or repeatable. Lack of training. Testing is difficult ...
LogiGear Magazine, December 2015: Test Automation

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe