Allpairs, Pairwise, Combinatorial Analysis

Last week I went to StarWest as a presenter and as a track chair to introduce speakers. Being a track chair is wonderful because you get to interface more closely with other speakers. Anyway…one of the speakers I introduced was Jon Bach. Jon is a good public speaker, and I was pleasantly surprised that he was doing a talk on the allpairs testing technique (also known as pairwise or combinatorial analysis). I wish Jon dedicated a little more time to the specifics of the technique during his talk and was generally more aware of available tools and information for folks to investigate further, but I think he successfully raised the general awareness and interest in pariwise testing as an effective testing technique among the audience.

Pairwise testing is one approach to solving the potential explosion in the number of tests when dealing with multiple parameters whose variables are semi-coupled or have some dependency on variable states of other parameters. For example, in the font dialog of MS Word there are 11 checkboxes for various effects such as superscript, strikethrough, emboss, etc. Obviously these effects have impact on how the characters in a particular font are displayed and can be used in multiple combinations such as Strikethrough + Subscript + Emboss. The total number of combinations of effects is the Cartesian product of the variables for each parameter, or 211 or 2048 in this example. This doesn’t include different font types, styles, etc. which also interdependent. So, you can see how the number of combinations increases rapidly especially as additional dependent parameters are included in the matrix.

The good news is the industry has a lot of evidence to suggest that most software defects occur from simple interactions between the variables of 2 parameters. So, from a risk based perspective where it may not be feasible to test all possible combinations how do we choose the combinations out of all the possibilities? Two common approaches include orthogonal arrays and combinatorial analysis.

But, true orthogonal arrays require that the number of variables is the same for all parameters. (Rarely true in software.) It is possible to create “mixed orthogonal arrays” where some combinations of variables will be tested more than once. For example, if we have 5 parameters and one parameter has 5 variables and the remains 4 parameters only have 3 variables each, we can see from the orthogonal array selector (available on FreeQuality website) the size of the orthogonal array is L25 (which basically means the test case will require 25 tests which is still significantly less than the total number of combinations of 405).

The other approach is combinatorial analysis (often referred to as pairwise or allpairs testing) because the approach most commonly used is to use a mathematical formula to reduce the total number of combinations in such a way that each variable for each parameter is tested with each variable from the other parameters at least once. In the above example, the number of tests would be reduced to 16. (Note: some tools will give slightly different results.) However, some tools (such as Microsoft’s PICT) also allow for more complex analysis of variable combinations such as triplets and n-wise coverage.

One problem that is hopefully not overlooked by testers using these tools is that some combinations of variables are simply not possible. For example, in the Effects group of the Font dialog it is impossible to check the Superscript checkbox and the Subscript checkbox simultaneously. Therefore, the tester either has to manually modify the output, or use a tool that allows constraints. Again, this is another situation where Microsoft’s free tool PICT excels. PICT uses a simply basic-like language for conditional and unconditional constraining of combinations of variables. PICT also allows weighting variables, seeding, output randomization, and negative testing.

I didn’t want this to be a PICT sales job, but alas my bias has influenced this post. So, I will conclude by pointing the readers to the Pairwise Testing website. My colleague Jacek Czerwonka has pulled together great resources on the technique of combinatorial analysis including a list of free and commercially available tools, and white papers supporting the value and practicality of this testing technique.

Article from Testing Mentor by BJ Rollison, Test Architect, Microsoft Inc


BJ Rollison, Software Test Architect for Microsoft.
Mr. Rollison started working for Microsoft in 1994, becoming one of the leading experts of test architecture and execution at Microsoft. He also teaches software testing courses for the University of Washington, and sits on the advisory board for testing certification at the University of Washington, the University of California Extension Santa Cruz, and Lake Washington Technical College.
BJ Rollison
BJ Rollison runs a successful consulting practice built on his more than twenty-five years of experience in the software industry. His career at Microsoft began on the Windows 95 team and ended on the Windows Phone team.

The Related Post

Introduction All too often, senior management judges Software Testing success through the lens of potential cost savings. Test Automation and outsourcing are looked at as simple methods to reduce the costs of Software Testing; but, the sad truth is that simply automating or offshoring for the sake of automating or offshoring will only yield poor ...
LogiGear Magazine March Issue 2021: Metrics & Measurements: LogiGear’s Guide to QA Reporting and ROI
PWAs have the ability to transform the way people experience the web. There are a few things we can agree we have seen happen. The first being that we figured out the digital market from an application type perspective. Secondly, we have seen the rise of mobile, and lastly, the incredible transformation of web to ...
Reducing the pester of duplications in bug reporting. Both software Developers and Testers need to be able to clearly identify any ‘Bug’, via the ‘Title’ used for the ‘Bug Report’.
Plan your Test Cases with these Seven Simple Steps What is a mind map? A mind map is a diagram used to visually organize information. It can be called a visual thinking tool. A mind map allows complex information to be presented in a simplified visual format. A mind map is created around a single ...
LogiGear Magazine March Testing Essentials Issue 2017
Regardless of the method you choose, simply spending some time thinking about good test design before writing the first test case will have a very high payback down the line, both in the quality and the efficiency of the tests. Test design is the single biggest contributor to success in software testing and its also ...
This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives. Article Synopsis There are many misconceptions about Software Testing. This article deals with the 5 most common misconceptions about how Software Testing differs from other testing. Five Common Misconceptions Some of the most common misconceptions about ...
LogiGear Magazine – February 2014 – Test Methods and Strategies
Creative Director at the Software Testing Club, Rob Lambert always has something to say about testing. Lambert regularly blogs at TheSocialTester where he engages his readers with test cases, perspectives and trends. “Because It’s Always Been Done This Way” Study the following (badly drawn) image and see if there is anything obvious popping in to ...
Experience-based recommendations to test the brains that drive the devices In essentially every embedded system there is some sort of product testing. Typically there is a list of product-level requirements (what the product does), and a set of tests designed to make sure the product works correctly. For many products there is also a set ...
Introduction Keyword-driven testing is a software testing technique that separates much of the programming work of test automation from the actual test design. This allows tests to be developed earlier and makes the tests easier to maintain. Some key concepts in keyword driven testing include:

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news