Avoid Epic Fail. Get Professional Help

Bringing in experts can set you up for automation success.

Test automation isn’t easy when your testing gets beyond a few hundred test cases. Lots of brilliant testers and large organizations have, and continue to struggle with test automation, and not for lack of effort. Everyone understands the value of test automation, but few testing organizations realize its full potential.

A year-long IDT (Innovative Defense Technologies) automated software testing survey showed that automated software testing fails to deliver the expected value for the follow reasons:

  • Lack of time: 37%
  • Lack of expertise: 20%
  • Lack of budget: 17%
  • Tool incompatibility: 11%
  • Other (mix of the above, etc.): 15%

Source: Implementing Automated Software Testing by Elfriede Dustin, Thom Garrett and Bernie Gauf

It’s not surprising that lack of time and lack of expertise are cited as the top reasons test automation fails. Test teams often get a tool thrust on them by non-QA decision makers- from a purchasing group that has no idea about testing, or a CIO who was influenced by a tool salesman, their previous company or, something from a conference. These non-QA people do not understand the coding or scripting skill level needed by the team, or may not realize the tool is harder to use than expected, or worse, the tool does not easily work with the application or system under test. As a result , the automation program languishes and then fails. And it’s the test team that looks bad.

If you’re starting a test automation program for the first time, or are charged with making testing faster and better a lot of the challenges will be the same. Test automation success begins at a high level. This is where the lack of (automation) expertise can cripple a project before it gets started.

Many people have unrealistic expectations about how easy or how quickly a program can ramp up and fail to take into account the impact on test schedules. Commitment to time and training and cooperation with developers must come from the highest levels in the organization. Having an outside expert share experiences and educate managers and directors about the complexities and realities of automation programs can set the right expectations and set you up for success by providing:

  • Strategic consulting— assess your current practices and create the strategy for your automation program.
  • Automation Consulting—develop and prioritize the automation effort and supporting activity specific to your needs.
  • Training— instruction in tools, processes and methods.
  • Coaching—hands-on guidance that helps teams become proficient faster.
  • What follows is where organizations can benefit from bringing in a qualified partner to help with their test automation.
Creating the Automation Strategy

Test automation is software development. It needs to be treated like its own project running parallel to production code development, with its own schedule, resourcing and project management. You need commitment and support from the top of the organization for a significant automation suite build-up.

Bringing in someone with prior success in automation implementation—a consultant—to assess your current practices and develop a custom-tailored plan will avoid making costly mistakes. A well-developed strategic plan will define the testing goal and outline the road map to achieve the goal. It will also establish an understanding of testing/quality assurance across the organization, and prescribe staff development to assure the team will have the necessary skills for test automation.

The strategy will guide other aspects of the automation effort, including tool selection. Typically tools are evaluated in this order: cost, ease of use, and features/functionality. There are low-cost easy to use tools— to a point— but getting the tool to do what is needed may require coding/scripting skills the team may not have. Also support is important. Support runs the gamut, from non-existent with open source tools to an included number of hours with commercial tools. The amount of support required will depend on the skills of the team, and it’s good to have when you run into a wall. Also, over-time, you application is likely to undergo quite a bit of change and having a tool that can be customized may be worth paying extra for. These considerations are generally not obvious to teams with little or no prior automation experience and this is where having a well-defined plan will avoid a costly long-term mistake.

Test Automation

This is the stage where the strategy gets implemented, but before just diving in and starting test automation, there’s a lot of work that needs to be done.

You’ll want to build infrastructure that includes naming conventions, standard or common coding practices, source control, error handling, managing data and reporting.

Teams should anticipate spending time to clean up existing tests before automating them. There are significant differences between tests written for manual execution and tests written for automation. Programmers typically don’t pay attention to “designing for testability”. They’re either unaware of how code changes impact automation or don’t effectively communicate changes. The result is that automation is negatively impacted. Failing to address test design issues will result in a large mass of test code that isn’t effective and requires a lot of time just to maintain. When teams understand good test design for automation it will become apparent there will be tests that should be thrown out and replaced with better written tests.

You will also need to develop and prioritize the automation. Taking 100 manual test cases and trying to create 100 automated tests will always be a bad idea. Not all tests are candidates for automation. If the application under test is not stable, it will impact which and how many tests can be automated. Then too, there will be manual tests that can and should be automated, and a lot that shouldn’t be.

Consultants with experience in successful automation implementations can help with all of these areas and enlisting their help at the beginning will get you started on the right track. Once you’re up and running it’s easy to build on success.

Training and Coaching

Training is the essential first step to gaining test automation expertise. There are a lot of good training resources. They range from free, on-demand recordings to generic-instructor led courses, to customized training. We’re seeing more and more companies opt for customized training as it’s not easy to translate generic concepts to specific applications.

The most important training for any team doing test automation, but especially those in Agile and Continuous Integration (CI) environments and teams doing large, high-volume automation, is learning test design. Up-front test design will determine how efficient tests are to run and how easy they are to maintain. Getting this right is what makes it possible to scale automation.

No matter where you are in your automation process, learning low-maintenance test design will enable you to get the most from automation. Action Based Testing (ABT) is a method of test design that lets you do this. It allows you to create common functions and function libraries that make for efficient test design and simplified maintenance. The limitation to training is that it only provides information. You have to turn the knowledge you receive into application, which can be slow, and you’ll probably make mistakes along the way. Getting hands-on instruction will shorten the learning curve dramatically.

Hands-on coaches are the equivalent of personal trainers for testing. Coaches are able to work 1-on-1, and with a small group on your actual testing projects. They can guide you in the test design and the correct approach specific to the application or system under test, and even sit at a keyboard and walk you through a process. Coaches not only help teams become proficient faster, they are also able to jump in and help resolve complex issues so the project keeps moving as expected.

Outsourcing Sense

Automation will not immediately reduce the test effort. In reality the effort typically increases when automation is undertaken. Building the testing infrastructure takes time and utilizes a lot of resources. Designing and creating new tests will also take time. The additional work load can overextend teams already taxed to keep up with current testing demands. So it’s no surprise that lack of time is the number one reason test automation fails.

Outsourcing some or all of the initial work makes a lot of sense. It will be done by people who have done it before. This saves time and avoids mistakes and gets everything integrated and provides you with properly designed tests from the start. Farming the work out can let you can keep up with existing testing, and then have a smooth transition to the new program. When you factor in everything, most of the time this will provide a better ROI than going the in-house route.

Summary

Test automation isn’t easy, but it can be made a lot easier. Expert consultants can assess the structure, staffing, the corporate quality goals, and provide exactly what you need for an effective test automation program. Going this route is not only smart, it will save time, produce long-term benefits, and most of all avoid epic fail.

Joe Luthy
Joe has over 15 years product marketing experience in technology and Internet services for major companies including AT&T and Verisign.TestArchitect for Visual Studio is a keyword authoring platform extension designed specifically to enhance coded UI test automation in Visual Studio 2012.

The Related Post

The path to continuous delivery leads through automation Software testing and verification needs a careful and diligent process of impersonating an end user, trying various usages and input scenarios, comparing and asserting expected behaviours. Directly, the words “careful and diligent” invoke the idea of letting a computer program do the job. Automating certain programmable aspects ...
A short-list of selection criteria and popular automation tools. There are a lot of test automation tools available in the market, from heavy-duty enterprise level tools to quick and dirty playback-and-record tools for browser testing. For anyone just starting their research we’ve put together a short list of requirements and tools to consider.
Introduction A common issue that I come across in projects is the relationship between test automation and programming. In this article I want to highlight some of the differences that I feel exist between the two.
Introduction A characteristic of data warehouse (DW) development is the frequent release of high-quality data for user feedback and acceptance. At the end of each iteration of DW ETLs (Extract-Transform-Load), data tables are expected to be of sufficient quality for the next ETL phase. This objective requires a unique approach to quality assurance methods and ...
TestArchitect TM is the name we have given to our automation toolset. It reflects the vision that automated testing requires a well-designed architectural plan allowing technical and non-technical elements to work fluidly in their capacity. It also addresses the continual missing link of all test automation tools of how to design tests. In TestArchitect the test ...
This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives. Introduction There are many potential pitfalls to Manual Software Testing, including: Manual Testing is slow and costly. Manual tests do not scale well. Manual Testing is not consistent or repeatable. Lack of training. Testing is difficult ...
Developers of large data-intensive software often notice an interesting — though not surprising — phenomenon: When usage of an application jumps dramatically, components that have operated for months without trouble suddenly develop previously undetected errors. For example, the application may have been installed on a different OS-hardware-DBMS-networking platform, or newly added customers may have account ...
Test automation can provide great benefits to the software testing process and improve the quality of the results…. but its use must be justified and its methods effective. The reasons to automate software testing lie in the pitfalls of manual software testing… As we all know too well, the average manual software testing program:
As I wrote in various articles, organization is one of the 3 key requisites for successful automated testing, the other two being test design and automation architecture.
Introduction Many executives have some very basic questions about Software Testing. These questions address the elements of quality (customer satisfaction) and money (spending the least amount of money to prevent future loss). The basic questions that executive have about Software Testing include: Why care about and spend money on testing? Why should testing be treated ...
Even the highest quality organizations have tradeoffs when it comes to their testing coverage. In Japan, Europe, and the United States, automotive manufacturers are aiming to enhance automotive functions by using software; in Japan in particular, Toyota, Nissan, Honda, Mazda, and Subaru are all adding endless amounts of software to their vehicles in the form ...
LogiGear Magazine – January 2011 – The Test Automation Issue

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe