Letter from the Editor – September 2018

As part of my work, I spend a lot of time at client’s sites and talk to various software development organizations. I am beginning to see a problem arise regarding Test Automation. There is too much automation! Surprised?

While there are still many teams struggling to make progress with Test Automation, many teams have been doing very significant Test Automation for a long, long time. Last year, we did a survey on Testing Essentials with our readership to figure out the current trends happening in our field. We discovered that 20% of respondents had no automation at all.

There has always been a lot of focus on organizations who are just starting on their Test Automation journey. There is good reason for this. They need the most help.

But, often times I think too little focus is given to companies that have been automating for a while who have big automation suites. As these automation programs continue to grow and mature, they need a different kind of help. In light of businesses trying to deliver faster than ever through Continuous Delivery and, even Continuous Deployment, long time automating businesses with overly large regression suites are possibly hindering their organization’s progress.

Continuous Delivery demands immediate feedback. When a team has been automating for a while, has a decent framework and some smart people; thousands of tests are easily attainable. When you run them against multiple VMs, configurations, devices or browsers, it can take days to get feedback. But, when the pipeline is automated, you think can run the full regression suite and let the team know what happens in 5 days? No.

This is a very common issue today that clearly can not continue. So what is a team supposed to do? The solutions are difficult, if not painful. We need to cut tests, clean up and clean out tests; remove duplicates in order to reduce the number of tests for faster runs. Cleaning out old, failing tests and duplicates is an easy idea but may take a long time to do. There’s a catch, cutting seemingly important tests to get faster runs but cutting coverage may be a political battle. We need to trace and tag tests to code and functions to streamline runs. But, cutting coverage increases risk. Team members and managers do not want to hear this.

The most important thing to remember at this point is that CD’s goal is for smaller changes with smaller releases. The idea is to roll out one release at a time, to keep it isolated, thereby limiting risk. Our automated suites must reflect this. It’s easy to visualize this idea using containers. If you swap out one container of code, do you have to run your 5 day full regression suite? Also remember, CD is based on significant, repeatable, formalized re-running of developer unit tests. The typically larger increase in unit level tests can offset the increased risk of cutting API or UI level tests.

Any company moving to CD knows there are new risks with speedier deployments. Risk should be well-known and communicated. Risk should be managed. Risk should be localized and small in scope, with easy roll-backs. There is a give and take here. It’s not only faster and faster, smaller deploys mean less risk. When I see an organization’s automation goal of “More, more, more!”, I tell them a better goal is to stay Lean and Mean.

Our Test Automation issue is one of the biggest and best read every year. That is a good thing. I hope there is useful information about Test Automation for every team- teams new to automation as well as long time automators. New ideas and help is a good thing for all of us.

In this issue we look in-depth at automation across a wide variety of spectrums. Our own Tu Nguyen and Christine Paras discuss How to Automate Oracle ERP Testing in the cover story. James Willett discusses Alan Richardson’s book, Automating & Testing a REST API, and Daniel Knott returns to discuss the mobile test pyramid. This issue of TestArchitect Corner discusses how TestArchitect integrates with HP Quality Center. Rounding up the lineup, Eran Kinsbruner discusses how to connect Test Automation to specific app features and devices. This issue is also filled with a resource section on automation videos for you to watch, and is great for newbies and advanced practitioners alike. We also have an infographic on Anti-Patterns and how to avoid them. Don’t forget to peruse our events Calendar. LogiGear will be at both STARWEST and STPCon-Fall. If you’re planning on attending these shows then stop by and say hello!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

A lot has changed since I began staffing test projects. From hiring college students and interns for summer testing programs, to building networks of offshore teams around the world, and from having 24-hour work schedules to having instant crowdsourced public beta or bug bounty testing—things have changed.
I have been excited about this issue since I included it in the 2011 editorial calendar. This issue of LogiGear Magazine dives into an exploration of agile automation—from the most efficient methods for test automation, to skill sets and better preparation for test teams, and even to understanding the variety of tools in question. We ...
As we settle into autumn, we’re taking the time to start some new traditions. This is LogiGear magazine’s first issue on SMAC. SMAC—social, mobile, analytics and cloud. We will be doing more issues in the next few years on these topics since so much of the product world is moving to this development stack.
The Greek philosopher Heraclitus of Ephesus (c. 500 BCE) is credited with saying, “The only constant is change.”   This is a statement that, more than 2,000 years later, still holds true. Today, we are in a time of great change. Everything is in flux. The fact is, we are always in a state of change even if ...
How do you test software? How do you validate it? How do you find bugs? These are all good questions anyone on your project team or anyone responsible for customers may ask you. Can you articulate your test strategy─not your test process, but explain your approach to testing? I find that this can be a ...
Testers need to learn their craft and hone in on their skill set. That means building skills, sharpening their tools, and becoming creative detectives. There is no cookie-cutter tester and no best practice. The best circumstance is a fully-skilled, aggressive tester mixed with curiosity, nimbleness, and agility.
Testing Embedded systems and testing the Internet of Things could each have their own issue of LogiGear magazine. But these days they are referred to presupposing knowledge of the other, so we thought it would be a good idea to tackle the two together in this issue to give a broad understanding of the landscape ...
Hello everyone – I’m hoping each one of us is having a great October. This time of the year is always my favorite, with the changing of the seasons, Fall was always my favorite time of year; it signified change and renewal – but I don’t want to digress to much from what’s going on ...
I was just recently at a company that had a beautiful test architecture, framework, and Cucumber with tons of well-automated tests. But there was no good test management on top of the Cucumber tests, and they did not do a good job tagging the tests. Although almost everybody on the team could write and maintain ...
If you are reading this issue, you are probably aware of the impact on the business world of cloud computing. Most people do not have a good grasp on what the cloud is or how people and products can use it. BTW, you are already a cloud user. If your email is stored somewhere “on ...
A while ago, I helped start a Software Quality Certificate Program as a part of the Software Engineering Program at the University of California, Santa Cruz Extension in Silicon Valley. I was on the Board of Advisors. While putting the curriculum together, a few people suggested a Measurement and Metrics course. Since I was teaching ...
In our continuing effort to be the best source of information for keeping testers and test teams current, we have another issue to explore testing in Agile development. As Agile evolves, systemic problems arise and common rough situations become apparent. We want to provide solutions. For anyone who has worked on Agile projects, especially if ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news