Making the Case for Better Test Design

The No-Nonsense Guide for How to Write Smarter and Low Maintenance Test Cases

TD1

Test design is a phrase that is often used when planning testing and test efforts, but I do not believe it is well understood. Also, opinions vary widely about the importance of test design ranging from irrelevant to the crucial ingredient for success.

Recently, I was at a company where they are throwing out their entire test automation suite and starting over. The regression suite they had been building for a few years was bloated and verbose. To make matters worse, the pass-rate (percentage of automated tests passing each run) kept dropping, and the team had long ago lost confidence that the regression suite even gave much assurance.

Their idea two years ago was to invest a bunch of money in a new tool and hire more technical staff. Next, take the manual test cases and automate them since they seem to work, and automating is better than manual. Good idea? Eighteen months later they had a large number of automated scripts, but the tests were poorly designed in the first place.

Relatively no thinking went into the scripts: the good testers used the scripts only as a guide to do more effective exploratory testing on the fly. Automating the manual tests lost that, and replaced it with boring, not useful, “happy path” automated tests. The tests were high maintenance, and some needed manual help to run.

The company achieved their goal of getting a “big” number of tests automated. The problem is the tests were not thoughtful tests. They missed breaks in the product, support calls went up, and confidence in the automation disappeared. Other Dev team members complained.

A reduced test team then focused only on testing new user stories. Their expertise was to have been transferred to automation through the test cases. That did not happen. And over the course of one year, most of the small number of automation engineers left the company — their work was regularly criticized.

Cost savings? Greater test coverage? Or greater confidence than manual testing? No.

The Company now has to start over. Again.

Clearly, a giant failure and waste of time, money and goodwill. There were many wrong ideas about testing in this situation. The first is not realizing the impact of design, or lack thereof. In this case, there was a complete lack of knowledge about test design.

Also, test automation is a journey, not a destination: you can be sure that you will need to update your scripts and code at least once. Designing for maintainability is key.

I’m sure many of you can tell horror stories along these lines.

The answer to these problems? The need for better-designed, smarter, lower-maintenance test design!

But what is it, you ask, and how do you do it?

What makes a test project successful?

Earlier I wrote that test design is the deciding factor between a successful and unsuccessful project.

TD2

Good Test Design has:

  • Effective test cases that find bugs and issues early
  • Gives the ability to convey confidence in the test effort or measure coverage to team
  • Tests that are easy to write, read, review and maintain
  • Improved customer experience… (and if we aren’t improving that, what are we doing?)

Failed test design has:

  • Missed bugs and/or late bugs
  • No way to give the team confidence in the project
  • Verbose tests; too many details; take a long time to write; difficult to understand without deep product or technological knowledge
  • High maintenance cost for test cases especially when the tests are maintained by an engineer who did not created them originally.
  • Problems with automation, needing a separate project to automate badly designed tests

Failed test design, especially for automation projects have:

  • High maintenance costs for test author and future automation engineers
  • Wastes time and money on poorly designed scripts, or results in a lack of designed test scripts
  • Little to no value, giving a false sense of confidence
  • Missed bugs
  • Fragile tests—tests that break often or easily
  • Unwieldy suites of too many tests lacking quick value

How to Correct Failure

Just like software development, tests need planning, architecture, design, before writing the test themselves.

LogiGear Magazine

Test Development

1 – Test planning

  1. Strategy is about how things will get done.
  2. Test plans are about schedules, risk, coverage, scope—not test cases.
  3. Test strategy and plan together—these can be high level: think coverage, environments, platform and compatibility, among other things.

2 – Organization

  1. How to organize the project: test types, test suites, test models
  2. Module or folder structure is organization

3 – Test design—This is about authoring test cases.

What does good test design look like?

The test design is how you will get the testing done; designing the tests that are used to execute the testing project. It includes what tests you need to write to get the job accomplished, and how you will write them. Test authoring is the big production part, and it includes defining the action works or keywords, but not the low level verbose step-by-step test execution. It’s about what you are supposed to be testing, and how to drive data through the test. At LogiGear, we use various methods and techniques to do the authoring. We use Model-based testing, action-based testing, error guessing, choosing data by equivalence class partitioning, and boundary value analysis, just to name a few. You are, essentially, engineering your test.

Design the test so you can throw a bunch of data at it. Design for both impact and exposing bugs. While engineering the tests, you should also think about design for re-use and easy maintainability.

TD4

Should a team use a Standard?

Is it important to have everyone doing the same thing: writing test cases the same way? Yes! Here’s why:

First, it is too common a practice for test teams to lack deep training. It benefits the team to ensure everyone has the same skills foundation. Having an internal standard will make reviewing and auditing, sharing, maintaining, and re-factoring tests much easier. Some individuals struggle to document tests well. Having a set guideline always helps.

Guidelines help the team write tests in a similar style, but the real benefit is cutting maintenance costs. When a whole team follows one guideline, and there are functional changes anyone on the team can easily understand what to revise—faster. This solves the problem of the team not understanding a unique way a test was written, then updating it incorrectly or needing to start from scratch.

Training staff, especially later with the addition of new staff, will be easier with set standards and guidelines. Various tools can also guide teams to write tests in the same style, but not necessarily with any intelligence. This is where great test design comes in to play.

Authoring tests—The Heart of Test Design

Some tools will help, or force you into a structure, some will not. Grouping tests into modules or suites is essential. It’s also obvious. You do not want a mess of hundreds or thousands of tests randomly listed with little structure or traceability. That would seriously and negatively impact the execution and maintenance of the tests.

The testing goal of the modules needs to be obvious and of the proper magnitude: Not too big, not too small. Do a good and thoughtful job organizing!

When authoring tests, focus on different parts at different places in the process.

Test cases have evolved dramatically over the last decade. Most tests started out in a homegrown style ranging from difficult to organize or maintain spreadsheets to naïve test tools that focus on low-level high-maintenance tests. These styles lack intelligence and disregard maintenance costs. We have evolved to focus more on test intelligence, while always remembering to design for low maintenance. Test cases have evolved into project assets.

Test cases, however they look, must fit into rapid sprints, be highly reusable, have clear objectives that state the business intelligence or goal, and be written in a natural language that anyone familiar with the application can understand. These describe what you are testing, not necessarily how to test it. The key is to Start High Level.

  • Test ObjectivesThese specify the intention and goal of the test; what do you hope to find from the test. In some cases, this is the most effective part of the test to review for sign-off. The objective is the test idea or goal: the reason why you are doing the test. These convey the real meaning of the test.
  • Test Steps—The step-by-step are the highest-maintenance part of documented tests: not of great value. Avoid documenting the steps until the latest point possible.
  • Test Data—Often the most interesting part of a test. The most effective tests are boundary tests. Use Equivalence Class Partitioning for the most efficient tests. Interesting data gives interesting tests. Boring data gives boring tests.
  • Validation point—The pass/fail criteria, expected results, check, checkpoint. Use good and correct validation points. Make sure that what you think you’re testing is what you are actually testing. Validation points need to be reviewed.

Test Objective

For test objectives, the main goal is to convey the meaning of the test. Below are three sample syntax that are easy, common, easy to teach:

  • Action+Function+Operating condition
  • Cause and Effect (From Action Based Testing)
  • Given-When-Then (From Behavior Driven Development)
P8 GRAPHIC

Consider the following: the idea behind the test (the test objective) may not change but the step-by-step of how to get that task or workflow is executed many often change. So, focus on simple, intelligent meaningful test objectives; this is the first big step to low maintenance testing. Do not focus on the steps.

Setup and Tear Down

After the Test Objectives have framed the tests to be done, there is still more engineering to be thought through. Many tests need setup: Initialize the test, set the exact or correct situation, add preconditions where needed. Just as many tests need setup, many need to be torn down and cleaned up after running.

Many people suggest tests can be piggy-backed to optimize setup and tear down. Use good judgment, as many people will suggest isolating tests to be completely independent. It would be too simplistic to make a blanket statement here. Think about lowering or increasing maintenance costs. Think about the speed of execution. Think about avoiding cascading failures where one test fails and causes more tests to fail, not because there are issues, but only because a test is reliant on a previous test passing to be executed. Use your brain here.

TD5

Increasing levels of details

Detail the actions, steps, procedures and call scripts. Beware these do not become the early focus of the tests. Keep it low maintenance. Only add low-level details as needed. Practice the Lean principle of Just-in-Time, JIT, writing the steps later or not at all is a great cost savings.

A common question about test design is when. When should we write our tests? The answer is layered.

Very early in the process, you must define and write the test objectives, that is, establish what it is that you are going to test. This will help you size the testing task, as well as provide a non-traditional type of TDD (test driven development) in that defining what you will test early in the cycle will help developers ensure those tests will pass.

Test design is an ongoing process. At the beginning of a sprint, the test objectives can be authored and detailed. Add more, lower-level details just-in-time to lower maintenance. The actions can be defined, but at this point it is less effective to get low level step-by-step details defined since the function is not yet built, and the UI is not defined.

This refocusing away from test scripts is a big shift for some testers. Low-level details for tests should not be documented during the initial development when it stands a better chance of needing to be redesigned, thereby raising the maintenance costs.

Low Maintenance/High reusability – just like with code

What is design for low maintainability?

  • Make the tests easy to write and read. Use a natural, domain-specific language.
  • Start high-level and build detail over time as the functions get delivered.
  • Define actions or keywords, but not execution steps to exercise the task or workflow.
  • Don’t get lost in details early.

Agile

In this agile era it is sadly, too common a practice that test teams under pressure for time focus on low-level tests. They don’t take a step back to look at the bigger picture and see test design from a global perspective. Agile testing is still in development. It does not focus solely on low-level user story acceptance criteria. These tests are too granular to get a great customer experience for the user doing end-to-end scenarios and workflows. It is important to remember that testing early in a sprint is not the same as testing later in a sprint, or in future sprints. The tests have to change, increasing in complexity and size.

Manual and Automated

Some testers have fallen into a trap: Manual testing is somewhat forgiving, in a bad way. You can have bad test design and fix it on the fly, or fail to have thoughtful test design and wing it. Manual testing is a bit more forgiving for bad test design in a few ways:

  • You may think you can just use brute force testing by “throwing more bodies on it.” This is never a good idea.
  • People expect slow or mediocre test efforts.
  • Low expectations for manual test projects.

Automation is too expensive and high profile to fail. When it does fail (and most do), some test teams cannot recover respect or confidence. Test automation projects are not as forgiving for the following reasons:

  • They are higher profile and therefore have more visibility.
  • They have higher cost.
  • Expectations are higher (due to the reasons mentioned above).

It does not matter if you are going to execute the tests manually or automatically: The test design should be good and effective for both, regardless. To think otherwise is a mistake.

Summary

If I can stress one thing, it’s this: design good tests. It makes sharing, maintenance and automating easier.

Excellent test design is essential for confidence in your testing job, meaningful coverage, making sure tests are run correctly and finding bugs! I have shown many effective and quick practices to get tests designed with speed and accuracy, especially in a low-maintenance way to set you up for effective and hopefully, large-scale automation. Good test design is not as hard as some people think, and common sense guidelines will make it easy for the whole team to write great tests. Go do it!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

How to fit automated testing into scrum, and keep testers in sync with other teams One of the benefits of the approaches of agile projects is their friendliness towards testing. The testing activities, and the testers with it, are integrated into the teams, and testing and quality are redefined as team responsibilities. Automation nowadays is a must-have ...
Application development and delivery teams are under constant pressure to release quality features as quickly as possible. CIOs rate delivering applications faster, with higher quality and with strong control on application development as their key priorities. What’s more, supporting this type of agile environment is particularly complex to IT teams that are also tasked with supporting ...
To begin this article, it would be a good idea to, remember this key point: Agile Manifesto Value #1 Individuals and interactions over processes and tools Tools work at the service of people. People, particularly intelligent people, can never be slaves to tools. People talking to each other, working together and solving problems is much ...
Agile stresses instant and easy communication and is built on teams working efficiently together. This necessitates an open work space environment. A characteristic of an effective team is a high level of collaboration, making the physical work environment an important factor. Cubicles should be eliminated in favor of an open work space in an effort ...
Author Jonathan Rasmusson explains in his latest book how to successfully set-up, execute and deliver Agile projects. Download the excerpt below for “Chapter 7: Estimation The Fine Art of Guessing.” To read his interview in last month’s issue, please click on “Spotlight Interview: Jonathan Rasmusson” to read his views on the best practices for test ...
I have worked with testers on an Agile team before and it has worked very well for both the team and the customer. In my previous role at Bank of Ireland, testers who had come from a traditional testing background worked within our teams to help ensure we had quality deliverables at the end of ...
Michael Hackett sat down with FNC’s Chris Floyd to get his take on numerous Agile topics.
Video narrated by MICHAEL HACKETT – Certified ScrumMaster This is Part One of a Four Part Video on “New Roles for Traditional Testers in Agile Development” Michael shares his thoughts on “A Primer – New Roles for Traditional Testers in Agile” LogiGear Corporation LogiGear Corporation LogiGear Corporation provides global solutions for software testing, and offers ...
Agile is a philosophy focused on delivering constant value to customers incrementally and frequently, based on communication and feedback. These two ingredients are vital to a successful Agile recipe. Agile is no longer a buzzword or an unknown territory in the industry. Agile has progressed leaps and bounds the last few years and has matured to ...
If your Agile implementation is not about people, you’ve missed the boat! The most profound impact to becoming more Agile is happier teams! Agile manifesto Value #1: * Individuals and interactions over processes and tools Words like these do not show up in Waterfall or RUP SDLC process descriptions. Agile cannot get more basic than ...
Keeping an eye on the horizon in the testing world is an important part of staying in the game. Hans is no stranger to looking to the future with eyes wide and ears open. His expertise is what makes Hans valuable at the STARWEST Expo, which he recently delivered two talks to.
One of the challenges with building an application these days is the number of dependencies that application will actually have on other applications. Ideally in order to know how that application will actually perform, application developers would be able to test their application against the application it depends on running in production. The odds of ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe