Bringing in experts can set you up for automation success.
Test automation isn’t easy when your testing gets beyond a few hundred test cases. Lots of brilliant testers and large organizations have, and continue to struggle with test automation, and not for lack of effort. Everyone understands the value of test automation, but few testing organizations realize its full potential.
A year-long IDT (Innovative Defense Technologies) automated software testing survey showed that automated software testing fails to deliver the expected value for the follow reasons:
- Lack of time: 37%
- Lack of expertise: 20%
- Lack of budget: 17%
- Tool incompatibility: 11%
- Other (mix of the above, etc.): 15%
Source: Implementing Automated Software Testing by Elfriede Dustin, Thom Garrett and Bernie Gauf
It’s not surprising that lack of time and lack of expertise are cited as the top reasons test automation fails. Test teams often get a tool thrust on them by non-QA decision makers- from a purchasing group that has no idea about testing, or a CIO who was influenced by a tool salesman, their previous company or, something from a conference. These non-QA people do not understand the coding or scripting skill level needed by the team, or may not realize the tool is harder to use than expected, or worse, the tool does not easily work with the application or system under test. As a result , the automation program languishes and then fails. And it’s the test team that looks bad.
If you’re starting a test automation program for the first time, or are charged with making testing faster and better a lot of the challenges will be the same. Test automation success begins at a high level. This is where the lack of (automation) expertise can cripple a project before it gets started.
Many people have unrealistic expectations about how easy or how quickly a program can ramp up and fail to take into account the impact on test schedules. Commitment to time and training and cooperation with developers must come from the highest levels in the organization. Having an outside expert share experiences and educate managers and directors about the complexities and realities of automation programs can set the right expectations and set you up for success by providing:
- Strategic consulting— assess your current practices and create the strategy for your automation program.
- Automation Consulting—develop and prioritize the automation effort and supporting activity specific to your needs.
- Training— instruction in tools, processes and methods.
- Coaching—hands-on guidance that helps teams become proficient faster.
- What follows is where organizations can benefit from bringing in a qualified partner to help with their test automation.
Creating the Automation Strategy
Test automation is software development. It needs to be treated like its own project running parallel to production code development, with its own schedule, resourcing and project management. You need commitment and support from the top of the organization for a significant automation suite build-up.
Bringing in someone with prior success in automation implementation—a consultant—to assess your current practices and develop a custom-tailored plan will avoid making costly mistakes. A well-developed strategic plan will define the testing goal and outline the road map to achieve the goal. It will also establish an understanding of testing/quality assurance across the organization, and prescribe staff development to assure the team will have the necessary skills for test automation.
The strategy will guide other aspects of the automation effort, including tool selection. Typically tools are evaluated in this order: cost, ease of use, and features/functionality. There are low-cost easy to use tools— to a point— but getting the tool to do what is needed may require coding/scripting skills the team may not have. Also support is important. Support runs the gamut, from non-existent with open source tools to an included number of hours with commercial tools. The amount of support required will depend on the skills of the team, and it’s good to have when you run into a wall. Also, over-time, you application is likely to undergo quite a bit of change and having a tool that can be customized may be worth paying extra for. These considerations are generally not obvious to teams with little or no prior automation experience and this is where having a well-defined plan will avoid a costly long-term mistake.
Test Automation
This is the stage where the strategy gets implemented, but before just diving in and starting test automation, there’s a lot of work that needs to be done.
You’ll want to build infrastructure that includes naming conventions, standard or common coding practices, source control, error handling, managing data and reporting.
Teams should anticipate spending time to clean up existing tests before automating them. There are significant differences between tests written for manual execution and tests written for automation. Programmers typically don’t pay attention to “designing for testability”. They’re either unaware of how code changes impact automation or don’t effectively communicate changes. The result is that automation is negatively impacted. Failing to address test design issues will result in a large mass of test code that isn’t effective and requires a lot of time just to maintain. When teams understand good test design for automation it will become apparent there will be tests that should be thrown out and replaced with better written tests.
You will also need to develop and prioritize the automation. Taking 100 manual test cases and trying to create 100 automated tests will always be a bad idea. Not all tests are candidates for automation. If the application under test is not stable, it will impact which and how many tests can be automated. Then too, there will be manual tests that can and should be automated, and a lot that shouldn’t be.
Consultants with experience in successful automation implementations can help with all of these areas and enlisting their help at the beginning will get you started on the right track. Once you’re up and running it’s easy to build on success.
Training and Coaching
Training is the essential first step to gaining test automation expertise. There are a lot of good training resources. They range from free, on-demand recordings to generic-instructor led courses, to customized training. We’re seeing more and more companies opt for customized training as it’s not easy to translate generic concepts to specific applications.
The most important training for any team doing test automation, but especially those in Agile and Continuous Integration (CI) environments and teams doing large, high-volume automation, is learning test design. Up-front test design will determine how efficient tests are to run and how easy they are to maintain. Getting this right is what makes it possible to scale automation.
No matter where you are in your automation process, learning low-maintenance test design will enable you to get the most from automation. Action Based Testing (ABT) is a method of test design that lets you do this. It allows you to create common functions and function libraries that make for efficient test design and simplified maintenance. The limitation to training is that it only provides information. You have to turn the knowledge you receive into application, which can be slow, and you’ll probably make mistakes along the way. Getting hands-on instruction will shorten the learning curve dramatically.
Hands-on coaches are the equivalent of personal trainers for testing. Coaches are able to work 1-on-1, and with a small group on your actual testing projects. They can guide you in the test design and the correct approach specific to the application or system under test, and even sit at a keyboard and walk you through a process. Coaches not only help teams become proficient faster, they are also able to jump in and help resolve complex issues so the project keeps moving as expected.
Outsourcing Sense
Automation will not immediately reduce the test effort. In reality the effort typically increases when automation is undertaken. Building the testing infrastructure takes time and utilizes a lot of resources. Designing and creating new tests will also take time. The additional work load can overextend teams already taxed to keep up with current testing demands. So it’s no surprise that lack of time is the number one reason test automation fails.
Outsourcing some or all of the initial work makes a lot of sense. It will be done by people who have done it before. This saves time and avoids mistakes and gets everything integrated and provides you with properly designed tests from the start. Farming the work out can let you can keep up with existing testing, and then have a smooth transition to the new program. When you factor in everything, most of the time this will provide a better ROI than going the in-house route.
Summary
Test automation isn’t easy, but it can be made a lot easier. Expert consultants can assess the structure, staffing, the corporate quality goals, and provide exactly what you need for an effective test automation program. Going this route is not only smart, it will save time, produce long-term benefits, and most of all avoid epic fail.
