Who Cares About Test Plans? You Should!

Test plans have a bad reputation, and perhaps, they deserve it! There’s no beating around the bush. But times have changed. Systems are no longer “black boxes” where QA Teams are separated from design, input, and architecture.

Test teams are much more technically savvy and knowledgeable about their systems, beyond domain knowledge. This was an old goal of Test Plans— to ensure test teams were testing the right things the right way. The increased knowledge of the systems and greater involvement in system design changes the need and use of test plans. Also, test teams are only one single aspect of testing. Dev tests, Ops tests, UI/UX teams test, Product Owners (PO) test, Users test. To write a traditional test plan as if “testers” are the only guardians of quality is circa 1995.

Also, no one likes to write them, and no one likes to read them. It’s a special team where they have good meaning and good use.

For as long as I have worked in software development test plans have had a mixed reputation:

  • Test plans take a long time to write.
  • Nobody reads them.
  • They take too much time away from actual testing.
  • It’s all just copy and pasting from the last test plan.
  • Things change so much once the project starts. They are too hard to maintain.
  • Halfway through the project they become obsolete.

All these things can be true

We need to transform and modernize the test plan to fit with today’s testing.

The goal of this article is to find out which things are important to you and your team and make your own test plan, or at least do a test planning practice.

Figure 1 – A collection of information regarding different test plan styles.

We’ve known, for as long as there have been test plans, that the planning process is far more valuable than the document. In the planning process you figure out how you are going to get things done. You plan the testing effort. Who is doing what, when, where, on what environments, what tools are needed, what dependencies do we have… etc.

The test plan is essentially a record of the planning process. The test plan is a vehicle for communication of the planning and resolved issues.

In the planning process you need to figure out, for example:

  • What test cases am I going to write for whatever features I need to test?
  • How long is it going to take for me to execute them? How many times do I have to execute them? Once per release? Once for every build?
  • Are these tests better run manually once or automated and put in my smoke test?
  • What tests are in scope, and which ones are out of scope?
  • What are the highest priority workflows, and what are the most common work flows?
  • What is missing, gaps or contingent on other teams or delivery?

AGILE, SCRUM, AND LEAN: Do we Even Need Test Plans?

Agile has further complicated this difficult relationship between testers and test plans. Most importantly, with a significantly shorter release cycle time, the idea that there will be time set aside for testers to document their plans is rarely going to happen. Also, with the Agile manifesto principle, “Working software over comprehensive documentation”, most teams have dramatically cut project and product documentation.

Agile, Scrum and Lean changed the notion of a test plan. Everyone pretty much stopped writing test plans. By the book, Scrum has nothing called a Test Plan. If you are writing a test plan, it’s because your organization wants one—modern SDLC and development paradigms do not have this notion.

Lean Software Development (LSD), based on Lean Manufacturing concepts from The Toyota Way, has had a giant impact on testing but also on any documentation, particularly test plans. The Lean practices of “cut waste” and “just in time” (design and planning) challenge what had been considered and understood as practice of writing test plans.

Test plans are often considered a waste— a waste of time, a waste of effort, and meaningless. Old style test plans? Yes, often a waste. As TCS said:

“A Test Plan is a valuable tool to the extent that it helps you manage your testing project and find bugs. Beyond that, it is a diversion of resources.

Testing Computer Software, p. 205, 1998

This was a Lean idea before it was cool. It is not a new thought that traditional test plans can be a huge waste of time.

What is an IEEE or Formal Test Plan?

This style for a formal, traditional test plan went out the window for many reasons. Scrum has the idea of the cross functional team that breaks down silo teams. There is not merely one person or team responsible for testing; testing is everyone’s responsibility on a Scrum team.

Also many of the aspects of traditional test plans, for example, risks, costs, training, any special equipment needs (mobile devices, servers, tools) shifted from being the responsibility of a test lead or test manager to the Scrum Master or Product Owner.

When testing was its own silo and there was a handoff to a test team, there was a notion of scope, exclusions, and cross-impacts, that needed to be documented, and agreed upon. In Agile the discussion of all that and the responsibility for it, again, falls on the Scrum Master and Product Owner. These were more likely to be things discussed at a daily standup meeting than to be documented.

Are there times when teams need traditional test plans? Absolutely! Some easy examples:

  • For compliance to regulatory agencies or internal groups during an audit.
  • Contractual agreements that your test plan must go outside the company, for example, to clients, to localization vendors for teams.
  • If you publish APIs, you could have implementation guides/SDK including a test plan.

There are many, many reasons why a test plan would have to go outside your company. If this is the case, you will probably be relying on the suggested IEEE template of what the contents are of the test plan, all are from the regulating agency and all are from the team or company requiring the test plan. I would divert to them on what they want or need included in the test plan.

Traditional Test Plans and Modern Test Plans: What Needs to Get Planned?

One problem I very commonly see is that, in Agile, teams are too focused on low level individual user stories and miss the bigger picture. A test plan will help take you from a myopic focus on the individual user story to seeing the bigger picture, with all the testing ramifications and needs resulting from it. When test plans got thrown out the window, planning the test project got thrown out the window with it as well. If the goal of a test plan was simply to put a check in a box signifying yes the test plan was written …good! I am glad teams stopped writing those. If you do still need this style test plan make it super Lean.

Even before the Agile revolution, most teams knew a lot from traditional test plan templates could be left out. At times the traditional test plans did serve a purpose. But these test plans have to lose a lot of fat.

An often controversial section of traditional test plans was “entrance and exit criteria,” or milestone criteria. What was “milestone criteria” is now Definition of Done (DoD) and belongs to the whole team. Writing about the process and tools like the bug tracking system or test case manager? That has all gone away. A lot of waste has been cut!

Test Plans Basics Revisited

What is a Test Plan and Why do we Want One?

A Test Plan is… a document describing the scope, approach, resources, and schedule of intended testing activities. It defines test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning.

The ANSI/IEEE Standard 829-1983 for Software Test Documentation

This is the beginning of the items covered in your planning and your test plan, if you need a document. It works from the IEEE list.

  • Scope (higher level).
  • Approach (Strategy).
  • Resources (anything that costs: people, tools, training, hardware, device, cloud service rental, etc.).
  • It has to have a schedule.
  • Test Items (collateral, accompanying databases, maps, training, quick start, Help, How-to, templates, wizards, etc.).
  • Features to test/features not to test (lower level scope).
  • Tasks and who will do them (sometimes turned into Roles and Responsibilities).
  • Risks, Contingencies, Dependencies, Exclusions, Assumptions.

To point out the obvious— these items have very little directly to do with individual or sets of test cases. This is much more risk management. The foundation of a modern test plan is dealing with risk, not a list of test cases.

The test plan is in response to a PO’s Release Plan.

Distinguish Test Plans and Test Strategies

Most testers work on functions, features or products for a long time. It gets developed, next release it gets expanded and bugs are fixed, maybe an immediate patch thrown in, two releases later, some other feature gets updated which impacts the other feature, then a change in workflow means updating… you see this. What this means is I can write a Test Strategy for tasks I do every time we work on this function. The strategy can describe what you have to test.

A test strategy describes how something will be tested, at what level, integration, interaction, cross impacts, where and how the testing is automated. It could detail which environment and with what data the thing will be exercised. But, it is independent of who and when. It is independent of cost, such as getting some expensive cloud mobile device test time, and independent of delivery contingencies. What this means is you can write a test strategy and it will be independent of the immediate project you are working on. It will last for, perhaps, two years with little maintenance. The test plan will only detail anything unique or different from the test strategy and focus only on this release, time/schedule, risk, and contingencies/dependencies. It’s the difference between “every time we test this feature, we have to do x,y,z” (strategy) versus “this time we will focus on test set x but not y and z (plan). We have an integration change with another API in this release plan.” It creates a long-term, lower maintenance test strategy, which makes a small test plan.

Test Plan Styles

Today, as we now know, it’s the planning and communication of the testing job rather than a document that is most important.

For all of us who still need to write test plans, here are common ways that people write test plans these days:

  • By release: Whether a quarterly release to customers or an annual or one-time release, plans are most often built around a release to customers.
  • By sprint: Since sprints are so short, a test plan built around a sprint is more rare but can be just as effective focusing on scope, exclusions, cross-impact.
  • By product: Test plans can also be written cross-team, by product. These type plans would be best at catching integration, cross-impacts, contingencies/dependencies, scheduling and roles and responsibilities.
  • By user story: It has become more and more common for teams to write a test plan per user story. These would, of course, be short, lean, fast, and focused. The usefulness would depend on how well and thorough the user story itself is written.
  • By function: An individual function’s testing can be planned. In organizations where this is common, I see test plans that clarify levels of testing for functions, as in unit testing integrations, testing through whatever UI, and where the function fits in a workflow, as well as, which and how the tests will be automated to isolate and verify correct behaviors after integration.

I often get asked the question: Is a set of test cases a test plan?

The answer is no. The idea that a collection of test cases could be understood as a test plan was around software testing in the old days of HP-Test Director (QC/Quality Center then ALM) style test case managers.

In those days, a test team took a block of time, a lot of written documents, knowledge of the users and the system to write most, if not all, of their proposed tests.

This theory sounds fine but the reality is a much different story. From my experience, many people who test find it difficult to get the business, product, programmers around a table and review only a set of test cases and understand the test project ahead. Many people on the team would not understand the nuances of the differences in certain test and certainly wouldn’t take the time to review whole sets of tests that could be as big as 100’s or 1000’s of tests. People on the project should understand what is tested and what is not as well as the risks associated.

More importantly today, aside from reviews and understanding, I believe development teams have to trust that their testers know what they’re doing, that testing staff needs to be a skilled and trained and given the proper tools, guidance, and information about what to test (testing strategies). In other words, the test team must be trusted to be competent.

Final Tips

Moving from old-style to modern test plans:

  • For the integrity of the test team, please, anything you may have “copied and pasted” in old test plans must be left out. It’s a waste.
  • Make whatever documentation you do write really useful. Lean doesn’t mean stop documenting.
  • Always communicate and get agreement on special test types that are included or specifically excluded: usability, performance, security, cross platform, any sort of compliance, need to be discussed, planned, and if needed, documented.
  • The longer organizations have been “Agile” a certain problem is growing, particularly with multiple Scrum team releases. From a few teams on one release to 80 teams on one release, without test plans, and the time spent planning a test project, over time, gaps show up, communication breakdowns increase, cross impact and integration problems become more common.
  • When the team is so completely focused on the user stories in front of them— as they should— it is easy to forget how changes can impact other teams. Without an effective “scrum of scrums,” process gaps grow.

Test plans are a document of the planning process. Write your plan for testing and you are on the right track. New test plans have to be lean: Just the facts.

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

In software testing, we need to devise an approach that features a gradual progression from the simplest criteria of testing to more sophisticated criteria. We do this via many planned and structured steps, each of which brings incremental benefits to the project as a whole. By this means, as a tester masters each skill or area ...
The V-Model for Software Development specifies 4 kinds of testing: Unit Testing Integration Testing System Testing Acceptance Testing You can find more information here (Wikipedia): http://en.wikipedia.org/wiki/V-Model_%28software_development%29#Validation_Phases What I’m finding is that of those only the Unit Testing is clear to me. The other kinds maybe good phases in a project, but for test design it ...
One of the most dreaded kinds of bugs are the ones caused by fixes of other bugs or by code changes due to feature requests. I like to call these the ‘bonus bugs,’ since they come on top on the bug load you already have to deal with. Bonus bugs are the major rationale for ...
David S. Janzen – Associate Professor of Computer Science Department California Polytechnic State University, San Luis Obispo – homepage LogiGear: How did you get into software testing and what do you find interesting about it? Professor Janzen: The thing I enjoy most about computing is creating something that helps people. Since my first real job ...
March Issue 2020: Smarter Testing Strategies for The Modern SDLC
Plan your Test Cases with these Seven Simple Steps What is a mind map? A mind map is a diagram used to visually organize information. It can be called a visual thinking tool. A mind map allows complex information to be presented in a simplified visual format. A mind map is created around a single ...
D. Richard Kuhn – Computer Scientist, National Institute of Standards & Technology LogiGear: How did you get into software testing? What did you find interesting about it? Mr. Kuhn: About 10 years ago Dolores Wallace and I were investigating the causes of software failures in medical devices, using 15 years of data from the FDA. ...
In today’s mobile-first world, a good app is important, meaning an effective Mobile Testing strategy is  essential.  
Back from more training, I was up at a client in Bellevue and really enjoyed teaching a performance class to a world class testing organization. I found that the students were very receptive to many of the concepts and ideas that the class offers.
Differences in interpretation of requirements and specifications by programmers and testers is a common source of bugs. For many, perhaps most, development teams the terms requirement and specification are used interchangeably with no detrimental effect. In everyday development conversations the terms are used synonymously, one is as likely to mean the “spec” as the “requirements.”
From cross-device testing, to regression testing, to load testing, to data-driven testing, check out the types of testing that are suitable for Test Automation. Scene: Interior QA Department. Engineering is preparing for a final product launch with a deadline that is 12 weeks away. In 6 weeks, there will be a 1 week quality gate, ...
I’ve been reviewing a lot of test plans recently. As I review them, I’ve compiled this list of things I look for in a well written test plan document. Here’s a brain dump of things I check for, in no particular order, of course, and it is by no means a complete list. That said, if you ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news