Test plans have a bad reputation, and perhaps, they deserve it! There’s no beating around the bush. But times have changed. Systems are no longer “black boxes” where QA Teams are separated from design, input, and architecture.
Test teams are much more technically savvy and knowledgeable about their systems, beyond domain knowledge. This was an old goal of Test Plans— to ensure test teams were testing the right things the right way. The increased knowledge of the systems and greater involvement in system design changes the need and use of test plans. Also, test teams are only one single aspect of testing. Dev tests, Ops tests, UI/UX teams test, Product Owners (PO) test, Users test. To write a traditional test plan as if “testers” are the only guardians of quality is circa 1995.
Also, no one likes to write them, and no one likes to read them. It’s a special team where they have good meaning and good use.
For as long as I have worked in software development test plans have had a mixed reputation:
- Test plans take a long time to write.
- Nobody reads them.
- They take too much time away from actual testing.
- It’s all just copy and pasting from the last test plan.
- Things change so much once the project starts. They are too hard to maintain.
- Halfway through the project they become obsolete.
All these things can be true
We need to transform and modernize the test plan to fit with today’s testing.
The goal of this article is to find out which things are important to you and your team and make your own test plan, or at least do a test planning practice.
We’ve known, for as long as there have been test plans, that the planning process is far more valuable than the document. In the planning process you figure out how you are going to get things done. You plan the testing effort. Who is doing what, when, where, on what environments, what tools are needed, what dependencies do we have… etc.
The test plan is essentially a record of the planning process. The test plan is a vehicle for communication of the planning and resolved issues.
In the planning process you need to figure out, for example:
- What test cases am I going to write for whatever features I need to test?
- How long is it going to take for me to execute them? How many times do I have to execute them? Once per release? Once for every build?
- Are these tests better run manually once or automated and put in my smoke test?
- What tests are in scope, and which ones are out of scope?
- What are the highest priority workflows, and what are the most common work flows?
- What is missing, gaps or contingent on other teams or delivery?
AGILE, SCRUM, AND LEAN: Do we Even Need Test Plans?
Agile has further complicated this difficult relationship between testers and test plans. Most importantly, with a significantly shorter release cycle time, the idea that there will be time set aside for testers to document their plans is rarely going to happen. Also, with the Agile manifesto principle, “Working software over comprehensive documentation”, most teams have dramatically cut project and product documentation.
Agile, Scrum and Lean changed the notion of a test plan. Everyone pretty much stopped writing test plans. By the book, Scrum has nothing called a Test Plan. If you are writing a test plan, it’s because your organization wants one—modern SDLC and development paradigms do not have this notion.
Lean Software Development (LSD), based on Lean Manufacturing concepts from The Toyota Way, has had a giant impact on testing but also on any documentation, particularly test plans. The Lean practices of “cut waste” and “just in time” (design and planning) challenge what had been considered and understood as practice of writing test plans.
Test plans are often considered a waste— a waste of time, a waste of effort, and meaningless. Old style test plans? Yes, often a waste. As TCS said:
“A Test Plan is a valuable tool to the extent that it helps you manage your testing project and find bugs. Beyond that, it is a diversion of resources.“
Testing Computer Software, p. 205, 1998
This was a Lean idea before it was cool. It is not a new thought that traditional test plans can be a huge waste of time.
What is an IEEE or Formal Test Plan?
This style for a formal, traditional test plan went out the window for many reasons. Scrum has the idea of the cross functional team that breaks down silo teams. There is not merely one person or team responsible for testing; testing is everyone’s responsibility on a Scrum team.
Also many of the aspects of traditional test plans, for example, risks, costs, training, any special equipment needs (mobile devices, servers, tools) shifted from being the responsibility of a test lead or test manager to the Scrum Master or Product Owner.
When testing was its own silo and there was a handoff to a test team, there was a notion of scope, exclusions, and cross-impacts, that needed to be documented, and agreed upon. In Agile the discussion of all that and the responsibility for it, again, falls on the Scrum Master and Product Owner. These were more likely to be things discussed at a daily standup meeting than to be documented.
Are there times when teams need traditional test plans? Absolutely! Some easy examples:
- For compliance to regulatory agencies or internal groups during an audit.
- Contractual agreements that your test plan must go outside the company, for example, to clients, to localization vendors for teams.
- If you publish APIs, you could have implementation guides/SDK including a test plan.
There are many, many reasons why a test plan would have to go outside your company. If this is the case, you will probably be relying on the suggested IEEE template of what the contents are of the test plan, all are from the regulating agency and all are from the team or company requiring the test plan. I would divert to them on what they want or need included in the test plan.
Traditional Test Plans and Modern Test Plans: What Needs to Get Planned?
One problem I very commonly see is that, in Agile, teams are too focused on low level individual user stories and miss the bigger picture. A test plan will help take you from a myopic focus on the individual user story to seeing the bigger picture, with all the testing ramifications and needs resulting from it. When test plans got thrown out the window, planning the test project got thrown out the window with it as well. If the goal of a test plan was simply to put a check in a box signifying yes the test plan was written …good! I am glad teams stopped writing those. If you do still need this style test plan make it super Lean.
Even before the Agile revolution, most teams knew a lot from traditional test plan templates could be left out. At times the traditional test plans did serve a purpose. But these test plans have to lose a lot of fat.
An often controversial section of traditional test plans was “entrance and exit criteria,” or milestone criteria. What was “milestone criteria” is now Definition of Done (DoD) and belongs to the whole team. Writing about the process and tools like the bug tracking system or test case manager? That has all gone away. A lot of waste has been cut!
Test Plans Basics Revisited
What is a Test Plan and Why do we Want One?
A Test Plan is… a document describing the scope, approach, resources, and schedule of intended testing activities. It defines test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning.
The ANSI/IEEE Standard 829-1983 for Software Test Documentation
This is the beginning of the items covered in your planning and your test plan, if you need a document. It works from the IEEE list.
- Scope (higher level).
- Approach (Strategy).
- Resources (anything that costs: people, tools, training, hardware, device, cloud service rental, etc.).
- It has to have a schedule.
- Test Items (collateral, accompanying databases, maps, training, quick start, Help, How-to, templates, wizards, etc.).
- Features to test/features not to test (lower level scope).
- Tasks and who will do them (sometimes turned into Roles and Responsibilities).
- Risks, Contingencies, Dependencies, Exclusions, Assumptions.
To point out the obvious— these items have very little directly to do with individual or sets of test cases. This is much more risk management. The foundation of a modern test plan is dealing with risk, not a list of test cases.
The test plan is in response to a PO’s Release Plan.
Distinguish Test Plans and Test Strategies
Most testers work on functions, features or products for a long time. It gets developed, next release it gets expanded and bugs are fixed, maybe an immediate patch thrown in, two releases later, some other feature gets updated which impacts the other feature, then a change in workflow means updating… you see this. What this means is I can write a Test Strategy for tasks I do every time we work on this function. The strategy can describe what you have to test.
A test strategy describes how something will be tested, at what level, integration, interaction, cross impacts, where and how the testing is automated. It could detail which environment and with what data the thing will be exercised. But, it is independent of who and when. It is independent of cost, such as getting some expensive cloud mobile device test time, and independent of delivery contingencies. What this means is you can write a test strategy and it will be independent of the immediate project you are working on. It will last for, perhaps, two years with little maintenance. The test plan will only detail anything unique or different from the test strategy and focus only on this release, time/schedule, risk, and contingencies/dependencies. It’s the difference between “every time we test this feature, we have to do x,y,z” (strategy) versus “this time we will focus on test set x but not y and z (plan). We have an integration change with another API in this release plan.” It creates a long-term, lower maintenance test strategy, which makes a small test plan.
Test Plan Styles
Today, as we now know, it’s the planning and communication of the testing job rather than a document that is most important.
For all of us who still need to write test plans, here are common ways that people write test plans these days:
- By release: Whether a quarterly release to customers or an annual or one-time release, plans are most often built around a release to customers.
- By sprint: Since sprints are so short, a test plan built around a sprint is more rare but can be just as effective focusing on scope, exclusions, cross-impact.
- By product: Test plans can also be written cross-team, by product. These type plans would be best at catching integration, cross-impacts, contingencies/dependencies, scheduling and roles and responsibilities.
- By user story: It has become more and more common for teams to write a test plan per user story. These would, of course, be short, lean, fast, and focused. The usefulness would depend on how well and thorough the user story itself is written.
- By function: An individual function’s testing can be planned. In organizations where this is common, I see test plans that clarify levels of testing for functions, as in unit testing integrations, testing through whatever UI, and where the function fits in a workflow, as well as, which and how the tests will be automated to isolate and verify correct behaviors after integration.
I often get asked the question: Is a set of test cases a test plan?
The answer is no. The idea that a collection of test cases could be understood as a test plan was around software testing in the old days of HP-Test Director (QC/Quality Center then ALM) style test case managers.
In those days, a test team took a block of time, a lot of written documents, knowledge of the users and the system to write most, if not all, of their proposed tests.
This theory sounds fine but the reality is a much different story. From my experience, many people who test find it difficult to get the business, product, programmers around a table and review only a set of test cases and understand the test project ahead. Many people on the team would not understand the nuances of the differences in certain test and certainly wouldn’t take the time to review whole sets of tests that could be as big as 100’s or 1000’s of tests. People on the project should understand what is tested and what is not as well as the risks associated.
More importantly today, aside from reviews and understanding, I believe development teams have to trust that their testers know what they’re doing, that testing staff needs to be a skilled and trained and given the proper tools, guidance, and information about what to test (testing strategies). In other words, the test team must be trusted to be competent.
Final Tips
Moving from old-style to modern test plans:
- For the integrity of the test team, please, anything you may have “copied and pasted” in old test plans must be left out. It’s a waste.
- Make whatever documentation you do write really useful. Lean doesn’t mean stop documenting.
- Always communicate and get agreement on special test types that are included or specifically excluded: usability, performance, security, cross platform, any sort of compliance, need to be discussed, planned, and if needed, documented.
- The longer organizations have been “Agile” a certain problem is growing, particularly with multiple Scrum team releases. From a few teams on one release to 80 teams on one release, without test plans, and the time spent planning a test project, over time, gaps show up, communication breakdowns increase, cross impact and integration problems become more common.
- When the team is so completely focused on the user stories in front of them— as they should— it is easy to forget how changes can impact other teams. Without an effective “scrum of scrums,” process gaps grow.
Test plans are a document of the planning process. Write your plan for testing and you are on the right track. New test plans have to be lean: Just the facts.
