Pairwise for API Testing: A CI/CD Friendly Model Based Automation

An approach to secure maintainability and scalability for API testing projects

One of the most important tasks in API testing is designing the input data whose quantum could explode due to the combination of a vast number of arguments. An experienced tester will inevitably figure out the advantages of the pairwise method in efficiently picking out a bundle of effective combinations.

However, testers will encounter pain points in maintaining and scaling a big pairwise testing project. This article proposes an approach to building up models for the APIs under test, generating test cases based on those models and actually executing those test cases on the real APIs. Additionally, we also guide you through the integration of this process into a CI/CD pipeline.

Don’t let APIs cause a test case explosion

As distributed and decentralized systems become a trend in the world of entrepreneurs, API designing and testing take a key role in cultivating quality services. A good design establishes a beautiful and friendly API whilst good tests ensure all quality aspects of a service including functionality, security, and performance. Both activities demand non-trivial experience, time and efforts. There are many tools that could support you in designing, and testing APIs as well as investigating the security and performance of those APIs.

With those tools in place, you may still find that testing an API is extremely expensive because of the large number of test cases. Unlike a legacy

function with only a few parameters, a modern API has dozens of parameters, each accepts a wide range of values. One single input usually conjoin into a lengthy XML or JSON document. Although one could apply common techniques such as equivalence partitioning (EP) and boundary values analysis (BVA) to reduce the number of test cases while preserving comparable defect-detection power, the combinatorial explosion when creating tests for a modern 20-something-parameter API is so enormous that one must find another (better) solution.

For example, a JIRA function named “create Issue” requires 31 parameters including project ID, type, summary, priority, assignee, due date, estimation, etc. Following EP and BVA, we could theoretically gather four values for each parameter and if exhaustively combined, we would test approximately four quintillions (billions of billion, 1018) of test cases.

On the other hand, a function call takes a few milliseconds whilst calling an API takes much longer (unless run locally) due to the network transmission. Executing thousands of API calls could take minutes or hours. Therefore, singling out the most effective test cases while discarding low-marginal-value tests cases is mandatory to achieve a minimal execution time.

Pairwise Testing is more similar to API Testing than you think

Pairwise method stems from the realization that most software failures are caused by a faulty single-factor input or by the combinatorial effect (interaction) of two factors. On top of that realization, we can also observe that progressively fewer and fewer failures are induced by interactions between three or more factors [1]. Pairwise testing is an approach in which all possible pairs of the parameter values are covered by at least one test. Although the pairwise method can take in both configuration parameters (describing the environment that the SUT will run in) and input parameters (describing the actual inputs that will be fed to the SUT), this article will only focus on input parameters since APIs are usually platform-agnostic.

As pairwise becomes a key technique in the testing industry, there are many good tools available to generate arrays with all pairs of parameter value combinations. Some of the popular tools are IPO [2], AETG [5], TCG and PICT [6]. In this article, we recommend IPO because of its simple model, amazing performance, and vigorous scalability.

Leverage EP and BVA techniques to bundle effectively

Pairwise testing may be regarded as a highly effective solution that solves the explosion and creates a bundle of effective test cases for API testing. Indeed, given a set of parameters and their values, a pairwise tool will generate an optimal set of test cases which is highly likely to detect defects.

However, applying pure pairwise tools would not be practical in API testing since those tools usually take in independent parameters whilst in reality, API parameters are almost always constrained by one another. For example, when testing the APIs from a flight booking system, the searching APIs should require that the return date must be after the departure date and that both should not be later than December 31st, 2018.

At LogiGear, we solve this problem by introducing a smart algorithm called Constraint-Based Elimination. Here’s how this Constraint-Based Elimination algorithm works:

  1. Leverages EP and BVA to create boundary combinations.
  2. The algorithm automatically generates tuples of the inter-dependent parameters as if those inter-dependent parameters are united into one conjoined parameter.
  3. All tuples that violate any predefined constraints will then be eliminated.
  4. Finally, the algorithm combines the conjoined parameters with the independent parameters and feed all of them to a traditional pair-wise library (in this case, IPO).

Thanks to this Constraint-Based Elimination algorithm, you can now reuse traditional pair-wise tools to generate test data for modern API tests given that inter-dependent parameters exist.

End-to-End Case

To illustrate our points, we’d like to walk you through an end-to-end use case.

First, modeling an API is required to automatically generate test cases for that API. Overall, an API model specifies the API call name including the base URI, method name, parameter names, types, and constraints. For convenience, we recommend modeling an API into two files, a WADL file for API specification and an XSD file for parameter specification and describing constraints.

Below is an example of modeling an API named ‘bookTicket’:

The picture below visually conveys the parameters and their constraints.

Once the model of an API is defined, you can start the Constraint-Based Elimination algorithm and pairwise algorithm to generate an effective set of test cases for that API. In order to achieve a full automation in API testing, you can use a unique tool called the Pairwise Test Data Generator. This tool will:

  • Load API model from WADL and XSD

Prepare the parameters automatically after considering the predefined constraints Apply Pairwise algorithm (IPO) to generate test cases Export generated test cases to a JSON file that is consumable by most test tools

Generally speaking, this Pairwise Test Data Generator will save you a significant cost in maintaining and scaling your test project. If the API somehow changes, you only need to modify the model (WADL and XSD). There is no need to modify the test flows or other test assets. Thanks to lower maintenance cost, your team is now free to focus on developing more tests and scale your project.

Ingredients to generate the API test cases

Once you execute the tool, you will get a JSON file, which contains all of the pairwise test cases that run faster than the exhaustive combinatorial approach but still possesses the same defect-detection power.

For more information on this LogiGear Pairwise Test Data Generator, please contact our Customer Success team. The below illustration shows the first two test cases generated by the tool.

Consume the Generated Test Data in TestArchitect

You can effectively use the above generated test data in any API test frameworks such as RestAssured. However, in the context of this article, we’ll illustrate how you consume the data using TestArchitect. Below is an example.

CI/CD pipeline integration

One of the most important advantages of our full-switched command line tool is its CI/CD readiness. The WADL and XSD files should be included as artifacts of a test and the tool should be pre-configured in the test environment. At that time, your CI/CD pipeline should define a pre-test step at which the Pairwise Test Data Generator tool is executed, and the generated test cases are saved to a JSON file within a given folder. The test tool, in turn, will load the JSON file before performing a data-driven test on the API under test.

In this article, we recommend using TestArchitect as a test tool in your CI/CD pipeline. With fully supported data-driven and API testing, TestArchitect is an ideal action-based testing tool for most CI/CD systems, to learn more about action based testing, you can look at the ABT infographic in this issue which explains it in-depth. Moreover, the tool officially supports running with Docker containers, enabling you to easily customize your pipeline despite the underlying CI/CD toolset.


In this article, we’ve proposed an approach based on pairwise testing and boundary combinations to generate effective test cases, particularly for API testing. Though the proposal covers most of the test cases for testing an API, it excludes negative cases as the original pairwise method does. That said, the combinatory testing method usually covers values of parameters despite their validity. Applying pairwise testing to create negative cases that combine invalid arguments is possible but it is not so practical since you will need to search for a solution to automatically generate the expected error handlings. It’s usually advisable to apply pairwise to happy-paths in parallel with manually designing test data for negative cases.

On the other hand, the proposal assumes that we have a small (limited) set of dependent parameters for a test. If the constraints are complicated, the algorithm may produce some complicated tuples combining several parameters. In the worst case, if all parameters are dependent, the boundary combination will take the key role and the IPO algorithm will do nothing. Thus, test coverage might be reduced.

Minh Ngo
Minh Ngo started his testing journey in 2008 as a Technical Engineer for Personal Navigation Devices at TomTom International B.V, Netherlands and joined LogiGear in 2012 as a Test Manager of TestArchitect product. He oversees LogiGear’s TCoE as Lead— responsibilities include R&D for automation, best practices, and testing services.
Thong Nguyen
Thong Nguyen joined LogiGear in 2009 as a software developer and got familiar with test automation. Thong currently is LogiGear’s lead in researching; he is responsible for setting up the innovation of TestArchitect, a leading tool in test automation. Nevertheless, Thong has not only joined many internal technical seminars, but has taught at the University of Sciences as an instructor in testing domain.

The Related Post

An API provides much of the functional capabilities in complex software systems. Most customers are accustomed to interacting with a graphical user interface on the computer. But, many do not realize that much of the functionality of a program comes from APIs in the operating system or the program’s dynamic-link libraries (DLL).
APIs are the unsung hero of our connected world We live in an exciting age of intelligence, where progress moves at the speed of imagination. We are connected to the world and one another like never before. API (Application Programming Interface) is the unsung hero of our connected world. Here’s everything you need to know ...
API testing has long been misunderstood as well-confined in the territory of developers. It’s natural to think that we must write code to test our code. However, it doesn’t have to be that way anymore. Business testers who have deep domain knowledge are now able to take on the challenges of API testing without coding. ...
An overview of web service testing solutions for traditional or non-technical testers. Much has been written on the technical execution of API tests, yet there are gaps in the details of what tests to design and how to design them. The articles tend to either get too technical too fast, or are too vague and ...
Lack of information and access to information isn’t an issue with web services. Web service documentation is widely available. Overview     One of the major persistent complaints from people who test is lack of information and lack of access to information. Luckily this is not the case with web services. If in the rare case with ...
APIs are subtly altering our expectations that there should be an app for everything. The concept of disruption has been given regal status across businesses, startups, and tech circles in recent years. With such great emphasis placed on change, user experiences are inevitably facing evolution as well. Application programming interfaces or APIs have great transformative powers to disrupt business, but are ...
API: An application programming interface (API) is a set of routines, protocols, and tools for building software applications. An API expresses a software component in terms of its operations, inputs, outputs, and underlying types. An API defines functionalities that are independent of their respective implementations, which allows definitions and implementations to vary without compromising the interface. Source:
These are the popular authentication methods in TestArchitect Authentication in API testing is usually a complicated subject for both developers and testers since it requires extensive knowledge on various types of security protocols and encryption algorithms.
 Understanding the pieces of the web service testing puzzle can make testing easier For people wanting a broader understanding of more pieces in the web service testing puzzle, here is a breakdown of the various possible components of an API.
API testing is different from GUI testing, but it doesn’t take long to master. What is an API? API is an acronym for Application Programming Interface. It enables communication and data exchange between two separate software systems. A software system implementing an API contains functions/subroutines which can be executed by another software system.
A case-study using: Java, REST Assured, Postman, Tracks, Curl and HTTP Proxies This is the first book review I have written on my site. So of course I had to choose a great book that was relevant to my niche. Alan Richardson’s book on Automating & Testing a REST API fits the bill perfectly. I am a big ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news