As part of my work, I spend a lot of time at client’s sites and talk to various software development organizations. I am beginning to see a problem arise regarding Test Automation. There is too much automation! Surprised?
While there are still many teams struggling to make progress with Test Automation, many teams have been doing very significant Test Automation for a long, long time. Last year, we did a survey on Testing Essentials with our readership to figure out the current trends happening in our field. We discovered that 20% of respondents had no automation at all.
There has always been a lot of focus on organizations who are just starting on their Test Automation journey. There is good reason for this. They need the most help.
But, often times I think too little focus is given to companies that have been automating for a while who have big automation suites. As these automation programs continue to grow and mature, they need a different kind of help. In light of businesses trying to deliver faster than ever through Continuous Delivery and, even Continuous Deployment, long time automating businesses with overly large regression suites are possibly hindering their organization’s progress.
Continuous Delivery demands immediate feedback. When a team has been automating for a while, has a decent framework and some smart people; thousands of tests are easily attainable. When you run them against multiple VMs, configurations, devices or browsers, it can take days to get feedback. But, when the pipeline is automated, you think can run the full regression suite and let the team know what happens in 5 days? No.
This is a very common issue today that clearly can not continue. So what is a team supposed to do? The solutions are difficult, if not painful. We need to cut tests, clean up and clean out tests; remove duplicates in order to reduce the number of tests for faster runs. Cleaning out old, failing tests and duplicates is an easy idea but may take a long time to do. There’s a catch, cutting seemingly important tests to get faster runs but cutting coverage may be a political battle. We need to trace and tag tests to code and functions to streamline runs. But, cutting coverage increases risk. Team members and managers do not want to hear this.
The most important thing to remember at this point is that CD’s goal is for smaller changes with smaller releases. The idea is to roll out one release at a time, to keep it isolated, thereby limiting risk. Our automated suites must reflect this. It’s easy to visualize this idea using containers. If you swap out one container of code, do you have to run your 5 day full regression suite? Also remember, CD is based on significant, repeatable, formalized re-running of developer unit tests. The typically larger increase in unit level tests can offset the increased risk of cutting API or UI level tests.
Any company moving to CD knows there are new risks with speedier deployments. Risk should be well-known and communicated. Risk should be managed. Risk should be localized and small in scope, with easy roll-backs. There is a give and take here. It’s not only faster and faster, smaller deploys mean less risk. When I see an organization’s automation goal of “More, more, more!”, I tell them a better goal is to stay Lean and Mean.
Our Test Automation issue is one of the biggest and best read every year. That is a good thing. I hope there is useful information about Test Automation for every team- teams new to automation as well as long time automators. New ideas and help is a good thing for all of us.
In this issue we look in-depth at automation across a wide variety of spectrums. Our own Tu Nguyen and Christine Paras discuss How to Automate Oracle ERP Testing in the cover story. James Willett discusses Alan Richardson’s book, Automating & Testing a REST API, and Daniel Knott returns to discuss the mobile test pyramid. This issue of TestArchitect Corner discusses how TestArchitect integrates with HP Quality Center. Rounding up the lineup, Eran Kinsbruner discusses how to connect Test Automation to specific app features and devices. This issue is also filled with a resource section on automation videos for you to watch, and is great for newbies and advanced practitioners alike. We also have an infographic on Anti-Patterns and how to avoid them. Don’t forget to peruse our events Calendar. LogiGear will be at both STARWEST and STPCon-Fall. If you’re planning on attending these shows then stop by and say hello!
