Who Cares About Test Plans? You Should!

Test plans have a bad reputation, and perhaps, they deserve it! There’s no beating around the bush. But times have changed. Systems are no longer “black boxes” where QA Teams are separated from design, input, and architecture.

Test teams are much more technically savvy and knowledgeable about their systems, beyond domain knowledge. This was an old goal of Test Plans— to ensure test teams were testing the right things the right way. The increased knowledge of the systems and greater involvement in system design changes the need and use of test plans. Also, test teams are only one single aspect of testing. Dev tests, Ops tests, UI/UX teams test, Product Owners (PO) test, Users test. To write a traditional test plan as if “testers” are the only guardians of quality is circa 1995.

Also, no one likes to write them, and no one likes to read them. It’s a special team where they have good meaning and good use.

For as long as I have worked in software development test plans have had a mixed reputation:

  • Test plans take a long time to write.
  • Nobody reads them.
  • They take too much time away from actual testing.
  • It’s all just copy and pasting from the last test plan.
  • Things change so much once the project starts. They are too hard to maintain.
  • Halfway through the project they become obsolete.

All these things can be true

We need to transform and modernize the test plan to fit with today’s testing.

The goal of this article is to find out which things are important to you and your team and make your own test plan, or at least do a test planning practice.

Figure 1 – A collection of information regarding different test plan styles.

We’ve known, for as long as there have been test plans, that the planning process is far more valuable than the document. In the planning process you figure out how you are going to get things done. You plan the testing effort. Who is doing what, when, where, on what environments, what tools are needed, what dependencies do we have… etc.

The test plan is essentially a record of the planning process. The test plan is a vehicle for communication of the planning and resolved issues.

In the planning process you need to figure out, for example:

  • What test cases am I going to write for whatever features I need to test?
  • How long is it going to take for me to execute them? How many times do I have to execute them? Once per release? Once for every build?
  • Are these tests better run manually once or automated and put in my smoke test?
  • What tests are in scope, and which ones are out of scope?
  • What are the highest priority workflows, and what are the most common work flows?
  • What is missing, gaps or contingent on other teams or delivery?

AGILE, SCRUM, AND LEAN: Do we Even Need Test Plans?

Agile has further complicated this difficult relationship between testers and test plans. Most importantly, with a significantly shorter release cycle time, the idea that there will be time set aside for testers to document their plans is rarely going to happen. Also, with the Agile manifesto principle, “Working software over comprehensive documentation”, most teams have dramatically cut project and product documentation.

Agile, Scrum and Lean changed the notion of a test plan. Everyone pretty much stopped writing test plans. By the book, Scrum has nothing called a Test Plan. If you are writing a test plan, it’s because your organization wants one—modern SDLC and development paradigms do not have this notion.

Lean Software Development (LSD), based on Lean Manufacturing concepts from The Toyota Way, has had a giant impact on testing but also on any documentation, particularly test plans. The Lean practices of “cut waste” and “just in time” (design and planning) challenge what had been considered and understood as practice of writing test plans.

Test plans are often considered a waste— a waste of time, a waste of effort, and meaningless. Old style test plans? Yes, often a waste. As TCS said:

“A Test Plan is a valuable tool to the extent that it helps you manage your testing project and find bugs. Beyond that, it is a diversion of resources.

Testing Computer Software, p. 205, 1998

This was a Lean idea before it was cool. It is not a new thought that traditional test plans can be a huge waste of time.

What is an IEEE or Formal Test Plan?

This style for a formal, traditional test plan went out the window for many reasons. Scrum has the idea of the cross functional team that breaks down silo teams. There is not merely one person or team responsible for testing; testing is everyone’s responsibility on a Scrum team.

Also many of the aspects of traditional test plans, for example, risks, costs, training, any special equipment needs (mobile devices, servers, tools) shifted from being the responsibility of a test lead or test manager to the Scrum Master or Product Owner.

When testing was its own silo and there was a handoff to a test team, there was a notion of scope, exclusions, and cross-impacts, that needed to be documented, and agreed upon. In Agile the discussion of all that and the responsibility for it, again, falls on the Scrum Master and Product Owner. These were more likely to be things discussed at a daily standup meeting than to be documented.

Are there times when teams need traditional test plans? Absolutely! Some easy examples:

  • For compliance to regulatory agencies or internal groups during an audit.
  • Contractual agreements that your test plan must go outside the company, for example, to clients, to localization vendors for teams.
  • If you publish APIs, you could have implementation guides/SDK including a test plan.

There are many, many reasons why a test plan would have to go outside your company. If this is the case, you will probably be relying on the suggested IEEE template of what the contents are of the test plan, all are from the regulating agency and all are from the team or company requiring the test plan. I would divert to them on what they want or need included in the test plan.

Traditional Test Plans and Modern Test Plans: What Needs to Get Planned?

One problem I very commonly see is that, in Agile, teams are too focused on low level individual user stories and miss the bigger picture. A test plan will help take you from a myopic focus on the individual user story to seeing the bigger picture, with all the testing ramifications and needs resulting from it. When test plans got thrown out the window, planning the test project got thrown out the window with it as well. If the goal of a test plan was simply to put a check in a box signifying yes the test plan was written …good! I am glad teams stopped writing those. If you do still need this style test plan make it super Lean.

Even before the Agile revolution, most teams knew a lot from traditional test plan templates could be left out. At times the traditional test plans did serve a purpose. But these test plans have to lose a lot of fat.

An often controversial section of traditional test plans was “entrance and exit criteria,” or milestone criteria. What was “milestone criteria” is now Definition of Done (DoD) and belongs to the whole team. Writing about the process and tools like the bug tracking system or test case manager? That has all gone away. A lot of waste has been cut!

Test Plans Basics Revisited

What is a Test Plan and Why do we Want One?

A Test Plan is… a document describing the scope, approach, resources, and schedule of intended testing activities. It defines test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning.

The ANSI/IEEE Standard 829-1983 for Software Test Documentation

This is the beginning of the items covered in your planning and your test plan, if you need a document. It works from the IEEE list.

  • Scope (higher level).
  • Approach (Strategy).
  • Resources (anything that costs: people, tools, training, hardware, device, cloud service rental, etc.).
  • It has to have a schedule.
  • Test Items (collateral, accompanying databases, maps, training, quick start, Help, How-to, templates, wizards, etc.).
  • Features to test/features not to test (lower level scope).
  • Tasks and who will do them (sometimes turned into Roles and Responsibilities).
  • Risks, Contingencies, Dependencies, Exclusions, Assumptions.

To point out the obvious— these items have very little directly to do with individual or sets of test cases. This is much more risk management. The foundation of a modern test plan is dealing with risk, not a list of test cases.

The test plan is in response to a PO’s Release Plan.

Distinguish Test Plans and Test Strategies

Most testers work on functions, features or products for a long time. It gets developed, next release it gets expanded and bugs are fixed, maybe an immediate patch thrown in, two releases later, some other feature gets updated which impacts the other feature, then a change in workflow means updating… you see this. What this means is I can write a Test Strategy for tasks I do every time we work on this function. The strategy can describe what you have to test.

A test strategy describes how something will be tested, at what level, integration, interaction, cross impacts, where and how the testing is automated. It could detail which environment and with what data the thing will be exercised. But, it is independent of who and when. It is independent of cost, such as getting some expensive cloud mobile device test time, and independent of delivery contingencies. What this means is you can write a test strategy and it will be independent of the immediate project you are working on. It will last for, perhaps, two years with little maintenance. The test plan will only detail anything unique or different from the test strategy and focus only on this release, time/schedule, risk, and contingencies/dependencies. It’s the difference between “every time we test this feature, we have to do x,y,z” (strategy) versus “this time we will focus on test set x but not y and z (plan). We have an integration change with another API in this release plan.” It creates a long-term, lower maintenance test strategy, which makes a small test plan.

Test Plan Styles

Today, as we now know, it’s the planning and communication of the testing job rather than a document that is most important.

For all of us who still need to write test plans, here are common ways that people write test plans these days:

  • By release: Whether a quarterly release to customers or an annual or one-time release, plans are most often built around a release to customers.
  • By sprint: Since sprints are so short, a test plan built around a sprint is more rare but can be just as effective focusing on scope, exclusions, cross-impact.
  • By product: Test plans can also be written cross-team, by product. These type plans would be best at catching integration, cross-impacts, contingencies/dependencies, scheduling and roles and responsibilities.
  • By user story: It has become more and more common for teams to write a test plan per user story. These would, of course, be short, lean, fast, and focused. The usefulness would depend on how well and thorough the user story itself is written.
  • By function: An individual function’s testing can be planned. In organizations where this is common, I see test plans that clarify levels of testing for functions, as in unit testing integrations, testing through whatever UI, and where the function fits in a workflow, as well as, which and how the tests will be automated to isolate and verify correct behaviors after integration.

I often get asked the question: Is a set of test cases a test plan?

The answer is no. The idea that a collection of test cases could be understood as a test plan was around software testing in the old days of HP-Test Director (QC/Quality Center then ALM) style test case managers.

In those days, a test team took a block of time, a lot of written documents, knowledge of the users and the system to write most, if not all, of their proposed tests.

This theory sounds fine but the reality is a much different story. From my experience, many people who test find it difficult to get the business, product, programmers around a table and review only a set of test cases and understand the test project ahead. Many people on the team would not understand the nuances of the differences in certain test and certainly wouldn’t take the time to review whole sets of tests that could be as big as 100’s or 1000’s of tests. People on the project should understand what is tested and what is not as well as the risks associated.

More importantly today, aside from reviews and understanding, I believe development teams have to trust that their testers know what they’re doing, that testing staff needs to be a skilled and trained and given the proper tools, guidance, and information about what to test (testing strategies). In other words, the test team must be trusted to be competent.

Final Tips

Moving from old-style to modern test plans:

  • For the integrity of the test team, please, anything you may have “copied and pasted” in old test plans must be left out. It’s a waste.
  • Make whatever documentation you do write really useful. Lean doesn’t mean stop documenting.
  • Always communicate and get agreement on special test types that are included or specifically excluded: usability, performance, security, cross platform, any sort of compliance, need to be discussed, planned, and if needed, documented.
  • The longer organizations have been “Agile” a certain problem is growing, particularly with multiple Scrum team releases. From a few teams on one release to 80 teams on one release, without test plans, and the time spent planning a test project, over time, gaps show up, communication breakdowns increase, cross impact and integration problems become more common.
  • When the team is so completely focused on the user stories in front of them— as they should— it is easy to forget how changes can impact other teams. Without an effective “scrum of scrums,” process gaps grow.

Test plans are a document of the planning process. Write your plan for testing and you are on the right track. New test plans have to be lean: Just the facts.

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

LogiGear Magazine March Issue 2018: Under Construction: Test Methods & Strategy
There are many ways to approach test design. These approaches range from checklists to very precise algorithms in which test conditions are combined to achieve the most efficiency in testing. There are situations, such as in testing mobile applications, complex systems and cyber security, where tests need to be creative, cover a lot of functionality, ...
First, let me ask you a few questions. Are your bugs often rejected? Are your bugs often assigned back to you and discussed back and forth to clarify information? Do your leaders or managers often complain about your bugs?
Introduction This 2 article series describes activities that are central to successfully integrating application performance testing into an Agile process. The activities described here specifically target performance specialists who are new to the practice of fully integrating performance testing into an Agile or other iteratively-based process, though many of the concepts and considerations can be ...
PWAs have the ability to transform the way people experience the web. There are a few things we can agree we have seen happen. The first being that we figured out the digital market from an application type perspective. Secondly, we have seen the rise of mobile, and lastly, the incredible transformation of web to ...
Having developed software for nearly fifteen years, I remember the dark days before testing was all the rage and the large number of bugs that had to be arduously found and fixed manually. The next step was nervously releasing the code without the safety net of a test bed and having no idea if one ...
Introduction Keyword-driven methodologies like Action Based Testing (ABT) are usually considered to be an Automation technique. They are commonly positioned as an advanced and practical alternative to other techniques like to “record & playback” or “scripting”.
Creative Director at the Software Testing Club, Rob Lambert always has something to say about testing. Lambert regularly blogs at TheSocialTester where he engages his readers with test cases, perspectives and trends. “Because It’s Always Been Done This Way” Study the following (badly drawn) image and see if there is anything obvious popping in to ...
March Issue 2019: Leading the Charge with Better Test Methods
The V-Model for Software Development specifies 4 kinds of testing: Unit Testing Integration Testing System Testing Acceptance Testing You can find more information here (Wikipedia): http://en.wikipedia.org/wiki/V-Model_%28software_development%29#Validation_Phases What I’m finding is that of those only the Unit Testing is clear to me. The other kinds maybe good phases in a project, but for test design it ...
People who follow me on twitter or via my blog might be aware that I have a wide range of interests in areas outside my normal testing job. I like to research and learn different things, especially psychology and see if it may benefit and improve my skills and approaches during my normal testing job. ...
When You’re Out to Fix Bottlenecks, Be Sure You’re Able to Distinguish Them From System Failures and Slow Spots Bottlenecks are likely to be lurking in your application. Here’s how you as a performance tester can find them. This article first appeared in Software Test & Performance, May 2005. So you found an odd pattern ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news