Picture a series of sprints: There are a variety of features being developed, with an eye towards having automated tests related to those features. Work starts to move along and Test Automation work likewise goes along with it. However, at some point, there invariably is that moment, usually in the middle of the project, where the need to do End-to-End Testing or some other aspect comes into play and there is now a need for a great deal of Manual Testing. Why? We have run into that specter I like to refer to as “The Midpoint Wall” or “The Midpoint Crisis.”
In short, if I am not careful or attentive at various stages, even with my best efforts, I can find myself in a situation where even having done my best and most vigilant work, I can be surrounded by a list of manual steps that now have to be performed. Have I been here? Certainly. The question, though, should be: “Why do I find myself here so frequently?”
What is a “Midpoint Crisis”?
First, before you furiously Google “Midpoint Crisis” to find out what I’m talking about, let me define it. You won’t find a definition for the “Midpoint Crisis” as it’s a term I (mostly) made up. However, it’s very similar to Steven Covey’s 7th Habit of “Sharpening the Saw.”
“Sharpening the Saw” essentially means to seek continuous improvement and renewal professionally and personally. How does that fit into the idea of the “Midpoint Crisis”? It’s the point where the odds of the idea of stepping back and sharpening the saw are the least appealing.
Here’s a mental image for you: You have a log that you want to saw. You have been at it for a while now, and you are about halfway through the log. You keep going when someone comes by and this is how the conversation goes:
Them: “How long have you been at this?”
You: “Hours!”
Them: “Wow, that’s a long time. What if you sharpen the saw?”
You: “I can’t do that silly, I’m too busy sawing!”
That’s the essence of the Midpoint Crisis: We are having to choose with doggedly pushing through manually or focusing on regression when, if we take the time to “sharpen the saw,” we might be able to do more and more effectively. Unfortunately, it’s hard to see that when we are at the midpoint of a project.
Now that you have a better idea of what a “Midpoint Crisis” is, let’s look at some common culprits that cause Midpoint Crises, and then see how we can manage through them.
Culprit 1: Testability
One of the areas that I find most frequently that leads to a Midpoint Crisis is the discovery of an area that makes for a difficult situation in regards to testability. One area that could be particularly frustrating is if you are dealing with dynamic data and, due to that data being changeable, you have elements that are generated at random. If I was aware of the fact that these elements would have been created randomly, I might have asked to see if there was another way that I could reach and set/unset or query them. Perhaps it is a process that is deeply nested in a variety of steps and menu transitions, or the information I need to get to requires traversing iFrames. Perhaps there is a level of necessary cross-server scripting that has to be anticipated.
I have found myself in this situation many times and ultimately it comes down to not knowing enough about the system or the necessary interactions before going in. The recommendation of course is to have a conversation with the team early on and determine what the testability options are and how we might reach them. To put it simply, we need to have this conversation sooner rather than later. The best time to talk about this is during the design workshop. If, however, you are in the middle of automating and you find that you have to switch gears because your automated tests will not work or the feature in question is resistant to Automation, the next best time to have that conversation is “now.”
As an example, I recently tried (with an emphasis on tried) to go in and attempt to navigate through a server and examine/set values needed in a deeply nested process. It required identifying a customer, finding their profile information, getting to their extended settings options, and then clicking through a variety of listings to find the right values and then enter them. Using a process, like WebDriver, made for long and drawn out interactions that were prone to breaking midway through the tests. I brought this up to the Developer and they said, “It’s not our server specifically so we have limited control over how to interact with it. However, I might be able to set up some code services that you can use and set those values directly through an API rather than this drawn out process.” With that, I was able to switch away from an extensive WebDriver Automation attempt (and frequent Manual Testing to fill in the gaps) by taking an alternate path. It’s important to recognize that that alternate path won’t be made unless we as Testers bring up the testability of what we are working with––don’t be afraid to ask!
Culprit 2: The End-to-End Maze
In simpler times, we could be expected to work on a standalone product and that product would have limited interactions with other systems. Unfortunately, for the most part, that is not the reality of many Testers today. Today’s systems are complex, have many moving parts, and as highlighted in my previous section, may require us interacting with systems and data that don’t originate with us or that are not necessarily under our control. Still, I have a need to test to make sure everything works and that often causes issues with Test Automation. Sure, I can automate my own areas but what do I do with those downstream systems? Can I reach them? Do I have access to interact with them? Can I change the data on their end to demonstrate the effect on my area of testing concern?
These are typical areas that I find to be both frustrating and inevitable. This can also be compounded when dealing with multiple systems set up in the cloud. When we have a system we are testing that requires interaction with other microservices hosted on other servers, and those other servers are part of different groups, the logistics often end up looking like a military campaign.
While mocking and stubs are helpful when it comes to development and doing unit tests, these will not help us when it is time to deploy and test these systems for real and with real data (not to mention at scale). This is where I encourage Testers, myself included, to develop what I like to call a “sphere of influence” map. This is a process where I diagram out all of the possible end-to-end interactions and with that, the servers/services I need to interact with. I reference this document in meetings and make sure that, if a new feature gets added, that I am clear if the feature touches on something within my sphere of influence. More importantly, I need to identify when it doesn’t. If a new microservice has been enabled on a new docker container set up on an EC2 server somewhere, I need to know if I can reach it and if my credentials are going to work. It is frustrating to find myself in the middle of work and having to ferret out what services are allowing me to interact and which ones aren’t and why. By identifying these early and doing the necessary diligence to make sure I can get to each service and interact, I can avoid a bottleneck later.
Culprit 3: Geography
How often have I worked on a test, manually walking through everything to make sure I have my workflow steps repeatable; yet, found that when automating, my tests aren’t working due to latency issues? This can be a real issue when it comes to geographically diverse regions and companies that hire people that work in a distributed manner.
Generally speaking, I write tests to work on my local system first, and then I run those tests on a variety of environments (we have QA, Staging, and Production machines set up for these scenarios). The challenge comes if I have to run from my own machine in my home office, connect to machines across the country (or in some cases across an ocean), and expect the tests to behave appropriately. Too often, they don’t. This can also be exacerbated with a hybrid cloud arrangement, where some services are in the cloud and other services are hosted by my company.
Over time, we have made this process cleaner by keeping our systems and their dependencies as geographically close as is practical. We have offices and customers all over the world and as such, that has meant that we have had to plan out clusters of machines in a variety of geographical areas to allow all theatres of operation the best performance possible. Likewise, when it comes to testing, if I have a geography issue, then I need to overcome that geographical challenge. It may work out that my development environment is only a first step and then all of my tests need to be packaged up and set up on a test server located in the geographical area where the servers are located. Likewise, my company has also made it possible to have servers in the cloud that are geographically located close to where I am so that when I am developing my tests, I do not have to deal with these geographical limitations as frequently. Once I am satisfied that my tests work, then I have the ability to export them to servers in different regions and run them there.
Culprit 4: Limitations to the Framework
I have been both fortunate enough to have a system that is well worked out and unfortunate enough to run into areas where nothing exists when it comes to new feature Automation work. Given time and attention, we can build out the needed Automation libraries to handle the new features and issues, but if we have that gap as a deadline is looming, then we are stuck with having to choose to automate or test. Sometimes, we just don’t get the option to do both.
In my company, we have automated tests and a code review for a feature to be considered “done” on the Scrum Board. Have I had to carry stories over to the next sprint because I couldn’t get it done in time? Sure. The good news is that having done that a number of times, it has allowed my team the opportunity to look and see what is missing. At times, I am able to fill in the gaps, but sometimes we need someone with more experience to come in (occasionally from another team if necessary) to help me get over that particular hurdle.
For those familiar with the “7 Habits” philosophies, this is the “Sharpen the Saw” step. Yes, it can be difficult to do the necessary work in the woodshed and mind the store at the same time. Still, my having these conversations to address the gaps help make these necessary steps achievable.
Conclusion
While it is not always possible to strike the perfect balance between Manual and Automated Testing, it helps to look at the various culprits listed above that can make my job more difficult than it needs to be. I want to emphasize that by no means is Manual Testing bad; there are still many great uses for Manual Testing like ad hoc or Exploratory Testing. However, the issue in regards to the Midpoint Crisis is that often these crises occur at times when ample time and resources haven’t been allotted to successfully complete the now-required Manual Testing (thus, the Midpoint Crisis). It would be great if we could always have enough time and bandwidth to go back and “sharpen our saw” in regards to our Automation, but you and I both know that’s not always possible; so, it’s important that we have the foresight to recognize possible crisis causes before we begin. By focusing on testability, an end-to-end strategy that makes sense, paying attention to how our infrastructure is configured, and occasionally getting help from others, we can keep these Midpoint Crises to a more manageable level.
Michael is a Black Belt in the Miagi-Do School of Software Testing, helped start and facilitate the Americas chapter of Weekend Testing, is a former Chair of the Education Special Interest Group with the Association for Software Testing (AST), a lead instructor of the Black Box Software Testing courses through AST, and former Board Member and President of AST. Michael writes the TESTHEAD blog and can be found on Twitter at @mkltesthead. A list of books, articles, papers, and presentations can be seen at http://www.linkedin.com/in/mkltesthead.
