In Agile/Safe Development, quality is sometimes sacrificed for scalability. Learn how to correctly scale your system without diminishing quality.

Over the past few years, I have had the opportunity to watch and participate in a variety of projects related to testing numerous systems. Over time these systems have grown to reflect various configurations designed to handle large numbers of users. For the growth in the front end, we allowed for load balancing of machines and dynamic addressing so that multiple machines would handle the increasing number of simultaneous connections. On the back end, we allowed for multiple machines to access our databases and to also replicate those databases to allow for increased transactions. As systems are developed to be allowed to scale, by their nature, they become more complex as more moving parts are put into play. These systems take on a new dynamic that all in one systems (most commonly what is used for day-to-day development work and testing) will never be able to approximate. This opens up the need to ask an important question. What do we prioritize, the scaling or the software quality? More to the point, do we necessarily have to make a choice?

What Challenges Come with Scalability?

Scalability comes down to how many users will be interacting with the system at any given time. How an organization gets to the point where they are able to maximize and leverage those capabilities is going to vary. In many cases, a tradeoff will need to be made. For systems that are going to be very large, and handle many concurrent connections, infrastructure and the ability to multiply a physical footprint is going to be the most important factor. This leads to a few questions:

• What does that look like?

• Does it mean that in-house replications of systems will be in order?

• Does it mean parallelizing systems and software components over multiple machines?

• Does it entail setting up large scale configurations of systems in the cloud to provide the necessary footprint to meet the needs of the growing organization?

It is tempting to think that, once the footprint and systems requirements are met, that most of the problem is solved. From my own experiences, and the development teams I have worked with, this is often misleading. With larger systems, more moving parts are required and those moving parts interact in ways that simpler all-in-one appliance systems do not. The desire to have a simple all-in-one system that runs on a development machine is understandable. Sure, it will be a decent platform to develop and examine the workflow and interactions that are expected for most scenarios, but it is likely to be limited to a handful of users or flows at any given time.

From a development perspective, even with Automation in the form of unit, integration, and end-to-end tests, odds are that most flows in this capacity would be run serially to make sure that they work. For initial development, feature checking, and proof of ideas with stub services, this makes sense. However, my experience shows that these systems and services are much less helpful when we need to crank up the number of concurrent transactions. What works for 5 interactions will not be sufficient to look at 50 simultaneous interactions, and certainly not 5,000 or 500,000 interactions.

Additionally, if an organization embraces Scaled Agile Framework principles (SAFe), we have to consider a number of factors:

• Who owns development? It is unlikely to be a single group but instead spread out through the entire organization.

• Who writes the modules (and who tests those modules) and at what level? Unit testing is going to be helpful for individual components and simple workflow transactions but will certainly not be sufficient to handling large amounts of connections, transactions, data, or any combination of the three. Additionally, who will handle the integration aspect of development between units?

• How will deployments work? What does a CI/CD pipeline look like for these types of large-scale applications? How is automated testing integrated? What level of load testing, performance testing, and security testing is put into place to examine the most risky areas for these setups?

These are not trivial questions, and oftentimes the solutions to adapting to larger projects like this adds to the complexity. Additionally, this is taking a monolithic view of an application, where we are considering just one aspect or section of functionality. From my own experiences, when an application scales, it doesn’t just scale upwards for the number of users or transactions. It also scales outward as it requires more pieces to hold everything together. Load balancers and more advanced network configurations come into play if you are stringing together multiple machines for a unified front end. Additionally, these systems are frequently dealing with not just a single application, but multiple applications needing to work together. Who will take on that integration piece? Will it be a matter of simply configuring one application in a large-scale environment, or will it need to take into account integration with other applications and services. Yes, a microservices architecture can help with this and unify the methods for interaction, to an extent, but they still need to be examined within a larger framework, they need to be proven to interact effectively and they need to all show that the system as a whole is working as intended in this larger environment.

Looking at Development in Larger Environments

One of the most challenging aspects of dealing with an application that needs to deal with a lot of interactions, a number of machines, and many moving parts is how those components interact with one another.

As mentioned previously, a single all-in-one appliance model may work fine for dealing with simple workflows or single user interactions. This will, however, be completely ineffective with a system that needs to be scaled up to handle a large number of transactions or users. The act of scaling systems means that each component added creates an additional layer of challenge. In addition to the architecture elements and how they fit together, we also need to consider which tests need to be configured to ensure that a load balancer is working correctly. Additionally, how do we ensure that timing will allow for database replication or parallel processing? How does a serial process compare to many parallel processes, and also, how do we interact with other groups and ensure their development efforts fit in with ours?

During the past several years, I had the opportunity to participate in these types of data transformation projects and some of the steps we had to do made for interesting and, at times frustrating, adaptations. Running tests serially was going to take too much time and the tools necessary to spin up large and robust environments would require a fairly sophisticated testing approach. Our solution at the time was to allow for parallel test runs, running simultaneously, and to focus on each system in the cluster. To that end, we created a system that had a manager node that would spin up several worker nodes and each of these worker nodes would either handle a subset of tests or allow for a spread of transactions on individual servers (creating parallel transactions to maximize the load balancing of a front end, or to effectively run queries or conduct transactions on designated databases in the back end.  

The more systems used in scaling the application surface, the more workers designated to run these transactions. Additionally, tests were created that worked in two dimensions. We were able to set up a large number of worker nodes and spread out our tests, meaning we could run a large number of unique tests simultaneously, thus cutting down the amount of time necessary to perform a build and deployment cycle. We could also replicate tests within these environments and shift the focus of testing to running similar tests in parallel, thus checking to see that we could handle a large number of concurrent transactions.

How Does This Affect Quality?

In my experience, the biggest challenge with balancing scalability and quality is the fact that the matrix of necessary things to examine grows larger and larger with each new component and expansion of the original application. To go back to my all-in-one appliance example (this is, by the way, the terminology we use for our simpler systems), it is possible for a small group to develop components, to automate basic workflows, and to create a series of repeatable and reliable tests to run in parallel with our manager and worker model for fast turnaround of a build to be deployed. In these larger scaled environments, that model breaks down quickly and the ability of a development team to handle all of the possibilities becomes impossible. While there is no truly exhaustive testing that can be performed (with the exception of trivial subsystems or atomic interactions) it is possible to make a risk assessment and focus on the areas that are the most critical. With systems built to larger scale, this risk assessment becomes more important and the need to focus on the critical areas takes precedence.

For these large scale systems, a divide and conquer approach is essential. Additionally, this is a scenario that requires an “all hands on deck” approach to development and testing, specifically when it comes to constructing a pipeline for integration and deployment. In an ideal situation, there would be small integrations performed regularly, so that individual components can be implemented. Additionally, large scale testing efforts would not have to be repeatedly performed.


When the goal is being able to make sure an application can run for a large number of users, or perform a large number of transactions simultaneously, the development and testing needs change and these changes can feel insurmountable. With a focus on full-team engagement, focusing on smaller areas of development and deployment, breaking down stories and deliverables in a way that allows for multiple deliveries, even applications that need to be built to scale can be tamed. Make no mistake, it will take time and it will take many hands working together to get to that point, but we need not sacrifice quality for the ability to scale. We have to change the way we look at quality and we may need to shift the goal posts to handle those changes and we may just need to be patient to allow for each side of the scale to be balanced.

Michael Larsen
Michael Larsen is a Senior Automation Engineer with LTG/PeopleFluent. Over the past three decades, he has been involved in software testing for a range of products and industries, including network routers & switches, virtual machines, capacitance touch devices, video games, and client/server, distributed database & web applications.

Michael is a Black Belt in the Miagi-Do School of Software Testing, helped start and facilitate the Americas chapter of Weekend Testing, is a former Chair of the Education Special Interest Group with the Association for Software Testing (AST), a lead instructor of the Black Box Software Testing courses through AST, and former Board Member and President of AST. Michael writes the TESTHEAD blog and can be found on Twitter at @mkltesthead. A list of books, articles, papers, and presentations can be seen at http://www.linkedin.com/in/mkltesthead.

The Related Post

One of the most common challenges faced by business leaders is the lack of visibility into QA activities. QA leaders have a tough time communicating the impact, value, and ROI of testing to the executives in a way that they can understand. Traditional reporting practices often fail to paint the full picture and do not ...
Introduction This article is part 2 of the 2-part series, Developer Testing? What is Testing and QA’s Place? Part 1 explored modern SDLCs, such as Agile, SCRUM, and Lean, specific to the dynamic of Developer Testing. It also discussed the traditional role of the Developer and the “testing” responsibilities typically delegated to them. The motivation ...
As organizations’ Agile methodologies become mature, one of the trends in software development is for Developers to take on more testing, and are eliminating the QA job function. In this article, written by Michael Hackett, we explore how Devs test, to the misconceptions about Devs & the SDLC in this 1st part of a 2-part ...
A lack of planning or foresight can doom Digital Transformation (DX). To provide the best customer experience (CX) possible, check out this strategic guide to effectively plan and implement your DX. Many organizations have sped up their processes in the name of Digital Transformation (DX). Part of these changes were business necessities brought about by ...
What is Continuous Testing? In today’s high-tech, modern world, innovation is driven by 3 things: speed, efficiency, and quality. This rings true across essentially every industry, and the Software Testing industry is no different. In order to stay competitive in a growing marketplace, software companies need to release their products faster, at a higher-quality, and ...
Has this ever happened to you: You’ve been testing for a while, perhaps building off of a branch, only to find out that, after all of this time, there is something big wrong. It’s a bad build and now you have to go backwards, fix something, and get a new build. Basically, you just wasted ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news