The No-Nonsense Guide for How to Write Smarter and Low Maintenance Test Cases
Test design is a phrase that is often used when planning testing and test efforts, but I do not believe it is well understood. Also, opinions vary widely about the importance of test design ranging from irrelevant to the crucial ingredient for success.
Recently, I was at a company where they are throwing out their entire test automation suite and starting over. The regression suite they had been building for a few years was bloated and verbose. To make matters worse, the pass-rate (percentage of automated tests passing each run) kept dropping, and the team had long ago lost confidence that the regression suite even gave much assurance.
Their idea two years ago was to invest a bunch of money in a new tool and hire more technical staff. Next, take the manual test cases and automate them since they seem to work, and automating is better than manual. Good idea? Eighteen months later they had a large number of automated scripts, but the tests were poorly designed in the first place.
Relatively no thinking went into the scripts: the good testers used the scripts only as a guide to do more effective exploratory testing on the fly. Automating the manual tests lost that, and replaced it with boring, not useful, “happy path” automated tests. The tests were high maintenance, and some needed manual help to run.
The company achieved their goal of getting a “big” number of tests automated. The problem is the tests were not thoughtful tests. They missed breaks in the product, support calls went up, and confidence in the automation disappeared. Other Dev team members complained.
A reduced test team then focused only on testing new user stories. Their expertise was to have been transferred to automation through the test cases. That did not happen. And over the course of one year, most of the small number of automation engineers left the company — their work was regularly criticized.
Cost savings? Greater test coverage? Or greater confidence than manual testing? No.
The Company now has to start over. Again.
Clearly, a giant failure and waste of time, money and goodwill. There were many wrong ideas about testing in this situation. The first is not realizing the impact of design, or lack thereof. In this case, there was a complete lack of knowledge about test design.
Also, test automation is a journey, not a destination: you can be sure that you will need to update your scripts and code at least once. Designing for maintainability is key.
I’m sure many of you can tell horror stories along these lines.
The answer to these problems? The need for better-designed, smarter, lower-maintenance test design!
But what is it, you ask, and how do you do it?
What makes a test project successful?
Earlier I wrote that test design is the deciding factor between a successful and unsuccessful project.
Good Test Design has:
- Effective test cases that find bugs and issues early
- Gives the ability to convey confidence in the test effort or measure coverage to team
- Tests that are easy to write, read, review and maintain
- Improved customer experience… (and if we aren’t improving that, what are we doing?)
Failed test design has:
- Missed bugs and/or late bugs
- No way to give the team confidence in the project
- Verbose tests; too many details; take a long time to write; difficult to understand without deep product or technological knowledge
- High maintenance cost for test cases especially when the tests are maintained by an engineer who did not created them originally.
- Problems with automation, needing a separate project to automate badly designed tests
Failed test design, especially for automation projects have:
- High maintenance costs for test author and future automation engineers
- Wastes time and money on poorly designed scripts, or results in a lack of designed test scripts
- Little to no value, giving a false sense of confidence
- Missed bugs
- Fragile tests—tests that break often or easily
- Unwieldy suites of too many tests lacking quick value
How to Correct Failure
Just like software development, tests need planning, architecture, design, before writing the test themselves.
Test Development
1 – Test planning
- Strategy is about how things will get done.
- Test plans are about schedules, risk, coverage, scope—not test cases.
- Test strategy and plan together—these can be high level: think coverage, environments, platform and compatibility, among other things.
2 – Organization
- How to organize the project: test types, test suites, test models
- Module or folder structure is organization
3 – Test design—This is about authoring test cases.
What does good test design look like?
The test design is how you will get the testing done; designing the tests that are used to execute the testing project. It includes what tests you need to write to get the job accomplished, and how you will write them. Test authoring is the big production part, and it includes defining the action works or keywords, but not the low level verbose step-by-step test execution. It’s about what you are supposed to be testing, and how to drive data through the test. At LogiGear, we use various methods and techniques to do the authoring. We use Model-based testing, action-based testing, error guessing, choosing data by equivalence class partitioning, and boundary value analysis, just to name a few. You are, essentially, engineering your test.
Design the test so you can throw a bunch of data at it. Design for both impact and exposing bugs. While engineering the tests, you should also think about design for re-use and easy maintainability.
Should a team use a Standard?
Is it important to have everyone doing the same thing: writing test cases the same way? Yes! Here’s why:
First, it is too common a practice for test teams to lack deep training. It benefits the team to ensure everyone has the same skills foundation. Having an internal standard will make reviewing and auditing, sharing, maintaining, and re-factoring tests much easier. Some individuals struggle to document tests well. Having a set guideline always helps.
Guidelines help the team write tests in a similar style, but the real benefit is cutting maintenance costs. When a whole team follows one guideline, and there are functional changes anyone on the team can easily understand what to revise—faster. This solves the problem of the team not understanding a unique way a test was written, then updating it incorrectly or needing to start from scratch.
Training staff, especially later with the addition of new staff, will be easier with set standards and guidelines. Various tools can also guide teams to write tests in the same style, but not necessarily with any intelligence. This is where great test design comes in to play.
Authoring tests—The Heart of Test Design
Some tools will help, or force you into a structure, some will not. Grouping tests into modules or suites is essential. It’s also obvious. You do not want a mess of hundreds or thousands of tests randomly listed with little structure or traceability. That would seriously and negatively impact the execution and maintenance of the tests.
The testing goal of the modules needs to be obvious and of the proper magnitude: Not too big, not too small. Do a good and thoughtful job organizing!
When authoring tests, focus on different parts at different places in the process.
Test cases have evolved dramatically over the last decade. Most tests started out in a homegrown style ranging from difficult to organize or maintain spreadsheets to naïve test tools that focus on low-level high-maintenance tests. These styles lack intelligence and disregard maintenance costs. We have evolved to focus more on test intelligence, while always remembering to design for low maintenance. Test cases have evolved into project assets.
Test cases, however they look, must fit into rapid sprints, be highly reusable, have clear objectives that state the business intelligence or goal, and be written in a natural language that anyone familiar with the application can understand. These describe what you are testing, not necessarily how to test it. The key is to Start High Level.
- Test Objectives—These specify the intention and goal of the test; what do you hope to find from the test. In some cases, this is the most effective part of the test to review for sign-off. The objective is the test idea or goal: the reason why you are doing the test. These convey the real meaning of the test.
- Test Steps—The step-by-step are the highest-maintenance part of documented tests: not of great value. Avoid documenting the steps until the latest point possible.
- Test Data—Often the most interesting part of a test. The most effective tests are boundary tests. Use Equivalence Class Partitioning for the most efficient tests. Interesting data gives interesting tests. Boring data gives boring tests.
- Validation point—The pass/fail criteria, expected results, check, checkpoint. Use good and correct validation points. Make sure that what you think you’re testing is what you are actually testing. Validation points need to be reviewed.
Test Objective
For test objectives, the main goal is to convey the meaning of the test. Below are three sample syntax that are easy, common, easy to teach:
- Action+Function+Operating condition
- Cause and Effect (From Action Based Testing)
- Given-When-Then (From Behavior Driven Development)
Consider the following: the idea behind the test (the test objective) may not change but the step-by-step of how to get that task or workflow is executed many often change. So, focus on simple, intelligent meaningful test objectives; this is the first big step to low maintenance testing. Do not focus on the steps.
Setup and Tear Down
After the Test Objectives have framed the tests to be done, there is still more engineering to be thought through. Many tests need setup: Initialize the test, set the exact or correct situation, add preconditions where needed. Just as many tests need setup, many need to be torn down and cleaned up after running.
Many people suggest tests can be piggy-backed to optimize setup and tear down. Use good judgment, as many people will suggest isolating tests to be completely independent. It would be too simplistic to make a blanket statement here. Think about lowering or increasing maintenance costs. Think about the speed of execution. Think about avoiding cascading failures where one test fails and causes more tests to fail, not because there are issues, but only because a test is reliant on a previous test passing to be executed. Use your brain here.
Increasing levels of details
Detail the actions, steps, procedures and call scripts. Beware these do not become the early focus of the tests. Keep it low maintenance. Only add low-level details as needed. Practice the Lean principle of Just-in-Time, JIT, writing the steps later or not at all is a great cost savings.
A common question about test design is when. When should we write our tests? The answer is layered.
Very early in the process, you must define and write the test objectives, that is, establish what it is that you are going to test. This will help you size the testing task, as well as provide a non-traditional type of TDD (test driven development) in that defining what you will test early in the cycle will help developers ensure those tests will pass.
Test design is an ongoing process. At the beginning of a sprint, the test objectives can be authored and detailed. Add more, lower-level details just-in-time to lower maintenance. The actions can be defined, but at this point it is less effective to get low level step-by-step details defined since the function is not yet built, and the UI is not defined.
This refocusing away from test scripts is a big shift for some testers. Low-level details for tests should not be documented during the initial development when it stands a better chance of needing to be redesigned, thereby raising the maintenance costs.
Low Maintenance/High reusability – just like with code
What is design for low maintainability?
- Make the tests easy to write and read. Use a natural, domain-specific language.
- Start high-level and build detail over time as the functions get delivered.
- Define actions or keywords, but not execution steps to exercise the task or workflow.
- Don’t get lost in details early.
Agile
In this agile era it is sadly, too common a practice that test teams under pressure for time focus on low-level tests. They don’t take a step back to look at the bigger picture and see test design from a global perspective. Agile testing is still in development. It does not focus solely on low-level user story acceptance criteria. These tests are too granular to get a great customer experience for the user doing end-to-end scenarios and workflows. It is important to remember that testing early in a sprint is not the same as testing later in a sprint, or in future sprints. The tests have to change, increasing in complexity and size.
Manual and Automated
Some testers have fallen into a trap: Manual testing is somewhat forgiving, in a bad way. You can have bad test design and fix it on the fly, or fail to have thoughtful test design and wing it. Manual testing is a bit more forgiving for bad test design in a few ways:
- You may think you can just use brute force testing by “throwing more bodies on it.” This is never a good idea.
- People expect slow or mediocre test efforts.
- Low expectations for manual test projects.
Automation is too expensive and high profile to fail. When it does fail (and most do), some test teams cannot recover respect or confidence. Test automation projects are not as forgiving for the following reasons:
- They are higher profile and therefore have more visibility.
- They have higher cost.
- Expectations are higher (due to the reasons mentioned above).
It does not matter if you are going to execute the tests manually or automatically: The test design should be good and effective for both, regardless. To think otherwise is a mistake.
Summary
If I can stress one thing, it’s this: design good tests. It makes sharing, maintenance and automating easier.
Excellent test design is essential for confidence in your testing job, meaningful coverage, making sure tests are run correctly and finding bugs! I have shown many effective and quick practices to get tests designed with speed and accuracy, especially in a low-maintenance way to set you up for effective and hopefully, large-scale automation. Good test design is not as hard as some people think, and common sense guidelines will make it easy for the whole team to write great tests. Go do it!
