An approach to secure maintainability and scalability for API testing projects
One of the most important tasks in API testing is designing the input data whose quantum could explode due to the combination of a vast number of arguments. An experienced tester will inevitably figure out the advantages of the pairwise method in efficiently picking out a bundle of effective combinations.
However, testers will encounter pain points in maintaining and scaling a big pairwise testing project. This article proposes an approach to building up models for the APIs under test, generating test cases based on those models and actually executing those test cases on the real APIs. Additionally, we also guide you through the integration of this process into a CI/CD pipeline.
Don’t let APIs cause a test case explosion
As distributed and decentralized systems become a trend in the world of entrepreneurs, API designing and testing take a key role in cultivating quality services. A good design establishes a beautiful and friendly API whilst good tests ensure all quality aspects of a service including functionality, security, and performance. Both activities demand non-trivial experience, time and efforts. There are many tools that could support you in designing, and testing APIs as well as investigating the security and performance of those APIs.
With those tools in place, you may still find that testing an API is extremely expensive because of the large number of test cases. Unlike a legacy
function with only a few parameters, a modern API has dozens of parameters, each accepts a wide range of values. One single input usually conjoin into a lengthy XML or JSON document. Although one could apply common techniques such as equivalence partitioning (EP) and boundary values analysis (BVA) to reduce the number of test cases while preserving comparable defect-detection power, the combinatorial explosion when creating tests for a modern 20-something-parameter API is so enormous that one must find another (better) solution.
For example, a JIRA function named “create Issue” requires 31 parameters including project ID, type, summary, priority, assignee, due date, estimation, etc. Following EP and BVA, we could theoretically gather four values for each parameter and if exhaustively combined, we would test approximately four quintillions (billions of billion, 1018) of test cases.
On the other hand, a function call takes a few milliseconds whilst calling an API takes much longer (unless run locally) due to the network transmission. Executing thousands of API calls could take minutes or hours. Therefore, singling out the most effective test cases while discarding low-marginal-value tests cases is mandatory to achieve a minimal execution time.
Pairwise Testing is more similar to API Testing than you think
Pairwise method stems from the realization that most software failures are caused by a faulty single-factor input or by the combinatorial effect (interaction) of two factors. On top of that realization, we can also observe that progressively fewer and fewer failures are induced by interactions between three or more factors [1]. Pairwise testing is an approach in which all possible pairs of the parameter values are covered by at least one test. Although the pairwise method can take in both configuration parameters (describing the environment that the SUT will run in) and input parameters (describing the actual inputs that will be fed to the SUT), this article will only focus on input parameters since APIs are usually platform-agnostic.
As pairwise becomes a key technique in the testing industry, there are many good tools available to generate arrays with all pairs of parameter value combinations. Some of the popular tools are IPO [2], AETG [5], TCG and PICT [6]. In this article, we recommend IPO because of its simple model, amazing performance, and vigorous scalability.
Leverage EP and BVA techniques to bundle effectively
Pairwise testing may be regarded as a highly effective solution that solves the explosion and creates a bundle of effective test cases for API testing. Indeed, given a set of parameters and their values, a pairwise tool will generate an optimal set of test cases which is highly likely to detect defects.
However, applying pure pairwise tools would not be practical in API testing since those tools usually take in independent parameters whilst in reality, API parameters are almost always constrained by one another. For example, when testing the APIs from a flight booking system, the searching APIs should require that the return date must be after the departure date and that both should not be later than December 31st, 2018.
At LogiGear, we solve this problem by introducing a smart algorithm called Constraint-Based Elimination. Here’s how this Constraint-Based Elimination algorithm works:
- Leverages EP and BVA to create boundary combinations.
- The algorithm automatically generates tuples of the inter-dependent parameters as if those inter-dependent parameters are united into one conjoined parameter.
- All tuples that violate any predefined constraints will then be eliminated.
- Finally, the algorithm combines the conjoined parameters with the independent parameters and feed all of them to a traditional pair-wise library (in this case, IPO).
Thanks to this Constraint-Based Elimination algorithm, you can now reuse traditional pair-wise tools to generate test data for modern API tests given that inter-dependent parameters exist.
End-to-End Case
To illustrate our points, we’d like to walk you through an end-to-end use case.
First, modeling an API is required to automatically generate test cases for that API. Overall, an API model specifies the API call name including the base URI, method name, parameter names, types, and constraints. For convenience, we recommend modeling an API into two files, a WADL file for API specification and an XSD file for parameter specification and describing constraints.
Below is an example of modeling an API named ‘bookTicket’:
The picture below visually conveys the parameters and their constraints.
Once the model of an API is defined, you can start the Constraint-Based Elimination algorithm and pairwise algorithm to generate an effective set of test cases for that API. In order to achieve a full automation in API testing, you can use a unique tool called the Pairwise Test Data Generator. This tool will:
- Load API model from WADL and XSD
Prepare the parameters automatically after considering the predefined constraints Apply Pairwise algorithm (IPO) to generate test cases Export generated test cases to a JSON file that is consumable by most test tools
Generally speaking, this Pairwise Test Data Generator will save you a significant cost in maintaining and scaling your test project. If the API somehow changes, you only need to modify the model (WADL and XSD). There is no need to modify the test flows or other test assets. Thanks to lower maintenance cost, your team is now free to focus on developing more tests and scale your project.
Once you execute the tool, you will get a JSON file, which contains all of the pairwise test cases that run faster than the exhaustive combinatorial approach but still possesses the same defect-detection power.
For more information on this LogiGear Pairwise Test Data Generator, please contact our Customer Success team. The below illustration shows the first two test cases generated by the tool.
Consume the Generated Test Data in TestArchitect
You can effectively use the above generated test data in any API test frameworks such as RestAssured. However, in the context of this article, we’ll illustrate how you consume the data using TestArchitect. Below is an example.
CI/CD pipeline integration
One of the most important advantages of our full-switched command line tool is its CI/CD readiness. The WADL and XSD files should be included as artifacts of a test and the tool should be pre-configured in the test environment. At that time, your CI/CD pipeline should define a pre-test step at which the Pairwise Test Data Generator tool is executed, and the generated test cases are saved to a JSON file within a given folder. The test tool, in turn, will load the JSON file before performing a data-driven test on the API under test.
In this article, we recommend using TestArchitect as a test tool in your CI/CD pipeline. With fully supported data-driven and API testing, TestArchitect is an ideal action-based testing tool for most CI/CD systems, to learn more about action based testing, you can look at the ABT infographic in this issue which explains it in-depth. Moreover, the tool officially supports running with Docker containers, enabling you to easily customize your pipeline despite the underlying CI/CD toolset.
Conclusion
In this article, we’ve proposed an approach based on pairwise testing and boundary combinations to generate effective test cases, particularly for API testing. Though the proposal covers most of the test cases for testing an API, it excludes negative cases as the original pairwise method does. That said, the combinatory testing method usually covers values of parameters despite their validity. Applying pairwise testing to create negative cases that combine invalid arguments is possible but it is not so practical since you will need to search for a solution to automatically generate the expected error handlings. It’s usually advisable to apply pairwise to happy-paths in parallel with manually designing test data for negative cases.
On the other hand, the proposal assumes that we have a small (limited) set of dependent parameters for a test. If the constraints are complicated, the algorithm may produce some complicated tuples combining several parameters. In the worst case, if all parameters are dependent, the boundary combination will take the key role and the IPO algorithm will do nothing. Thus, test coverage might be reduced.
