Automation and Requirements

Introduction

In many of the Test Automation projects that we are involved with using our Action-Based Testing methodology, management has expressed a need to relate tests and test results to system requirements. The underlying thought is that automation will create extra possibilities to control the level of compliance to requirements of the system under test. In this article I will explore some of the aspects and considerations regarding tests and requirements.

System and Test Requirements with Action-Based Testing

Action-Based Testing encourages testers to create “test requirements” as part of the test modules (spreadsheets with the tests to be carried out). The test requirements govern the scope of test cases that are in the test module; in other words, they detail the intention the tester has with the cases described.

For example, a system requirement is typically formulated like “the system should.” A test requirement typically reads as “test if.” As a short hand, we often leave the “test if” out, so a test requirement could read “the system does not accept password shorter than 6 characters,” to be understood as “test if the system does not accept password shorter than 6 characters.”

Now the next question is: how do the test requirements relate to system requirements? In many cases the link is very direct. As illustrated in the example above on minimum password length, the test requirement will relate one-on-one to a system requirement. In other cases a system requirement is a composite statement, which would lead to several test requirements.

For example “the system should enforce that passwords are at least 6 characters in length and contain at least 1 digit,” would lead to (1) “the system does not accept passwords shorter than 6 characters,” and (2) “the system requires a password to have at least 1 digit.”

Good Test Design: Going Beyond System Requirements

Another possibility is that a test requirement specifies that a test be created that does not directly relate to any system requirement. For example, a valid test requirement could state “test with 5 users from different states who are using the same mailbox.” This means the test should create a test case in which the described situation is simulated, but there is very likely no system requirement talking about 5 users.

Test requirements that are unrelated to system requirements illustrate an important consideration regarding system requirements and tests: A test designer should use system requirements as input and make sure all of them are covered in test cases, but a good test designer should go beyond that narrow mission. System requirements mostly focus on single elements of a system under test. A tester should make sure that the elements of the system work well together, and that the system can handle complex situations that the developer might not necessarily have thought about, because they were not specifically addressed in system requirements.

One potential benefit of Automation is to provide information related to system requirements. However, this needs to be carefully organized to be effective. Consider the following picture in which some relationships have been illustrated.

The solid arrows show how system requirements are, directly or indirectly, related to test requirements, which in turn are related to test cases, which in turn are related to test results. A test tool like TestArchitect will give you the possibility to establish relations 1-3 in the above picture as follows:

  1. In the details panel for a test requirement it is possible to register a “source”, this field can contain the identifier of the system requirement to which the test requirement relates.
  2. The same detail panel also shows the test cases in the test modules (after they have been created and the module has been checked in at least once), and checkmarks to indicate which one(s) cover the test requirement.
  3. The relation from the test results to the test case will be shown in the test results. Once the test result is checked in the test case details panel will show the latest result.

To easily obtain the consolidated relation (4) make sure the “source” fields of the test requirements are consistently populated, the test requirements are related to test cases and test run results (of successful runs) are checked in. This will make it easy to use the report generator in TestArchitect to show the test results per system requirement: define a report that shows test requirements, and as grouping field specify the “source” field.

Conclusion

This article has outlined that good test design has 3 major components:

  • Creating test cases that directly relate to system requirements
  • Creating test cases that indirectly relate to system requirements
  • Creating test cases that do not directly relate to system requirements

The third item is often overlooked and can be very critical in a successful Test Automation effort.

More Information

  • More information about Action-Based Testing may be found here.
  • More information on TestArchitect may be found here.
Hans Buwalda
Hans Buwalda, CTO of LogiGear, is a pioneer of the Action Based and Soap Opera methodologies of testing and automation, and lead developer of TestArchitect, LogiGear’s keyword-based toolset for software test design, automation and management. He is co-author of Integrated Test Design and Automation, and a frequent speaker at test conferences.

The Related Post

Automated Testing is a huge part of DevOps, but without human-performed quality assurance testing, you’re increasing the risk of  lower-quality software making it into production.  Automated Testing is an essential DevOps practice to increase organizations’ release cadence and code quality. But there are definitely limits to only using Automated Testing. Without human quality assurance (QA) ...
< Michael Hackett sat down with EA’s Stephen Copp to discuss the world of integrated test platforms.
For this interview, we talked to Greg Wester, Senior Member Technical Staff, Craig Jennings, Senior Director, Quality Engineering and Ritu Ganguly, QE Director at Salesforce. Salesforce.com is a cloud-based enterprise software company specializing in software as a service (SaaS). Best known for its Customer Relationship Management (CRM) product, it was ranked number 27 in Fortune’s 100 ...
Mobile usage today is not just a trend but it is an essential shift in how people communicate with each other, interact with the world, and do business. According to a ComScore, in 2014 the number of mobile users surpassed the number of computer users and is showing strong growth over time, towards some point in ...
Framework: An abstraction in which software providing generic functionality can be selectively changed by additional user written code, thus providing application specific software. A software framework is a universal, reusable software platform used to develop applications, products and solutions. Harness: A collection of software and test data configured to test a program unit by running it under varying conditions and monitoring ...
Test automation can provide great benefits to the software testing process and improve the quality of the results…. but its use must be justified and its methods effective. The reasons to automate software testing lie in the pitfalls of manual software testing… As we all know too well, the average manual software testing program:
Understanding the benefits and challenges of Automating ERP is critical. According to SAP, ERP (Enterprise Resource Planning) “is the core processes that are needed to run a company: finance, human resources, manufacturing, supply chain, services, procurement, and others. At its most basic level, ERP integrates these processes into a single system. But new ERP systems ...
I’ve been teaching a lot lately, was in India for one week, and I’m off to Seattle in two weeks to teach on performance topics. I thoroughly enjoy teaching, it allows me to stay sharp with current trends, and provides a nice break from the “implementation focus” that I generally have day to day.
Test execution and utility tools that can make your job easier My first exposure to the necessity for testers to have an array of tools was from the groundbreaking article “Scripts on my Toolbelt” by Danny Faught. Danny laid out the ideal approach to any testing job, and it got me thinking “How can I ...
All too often, software development organizations look at automating software testing as a means of executing existing test cases faster. Very frequently there is no strategic or methodological underpinning to such an effort. The approach is one of running test cases faster is better, which will help to deliver software faster. Even in organizations that ...
I feel like I’ve spent most of my career learning how to write good automated tests in an Agile environment. When I downloaded JUnit in the year 2000 it didn’t take long before I was hooked – unit tests for everything in sight. That gratifying green bar is near-instant feedback that everything is going as ...
LogiGear Magazine – September 2010

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe