Author of The Agile Warrior, Rasmusson answers questions from LogiGear’s testing staff about test automation in agile projects.
As an experienced entrepreneur and former agile coach for ThoughtWorks, Rasmusson has consulted internationally, helping others find better ways to work and play together. When not coaching his sons’ hockey teams or cycling to work in the throes of a Canadian winter, Jonathan can be found sharing his experiences with agile delivery methods at his blog, The Agile Warrior. Developer by trade, Jonathan Rasmusson engages with LogiGear’s inquisitive staff to give us insight on how he addresses various agile test automation challenges. An excerpt from chapter seven of his book, titled “Planning Agile Projects’ Estimation – The Fine Art of Guessing” will be included in next month’s issue focused on Test Process Improvement. More excerpts can be found at The Pragmatic Bookshelf.
1. How does test case automation fit in among different sprints? How can you automate testing on new functionality during a sprint? How do you make decisions during a spring to automate a test or not?
Generally when creating software in agile we like to do our testing in the same iteration (or sprint) that the story is developed in–this ensures that what we’ve produced works. It’s more efficient than abandoning the story and picking it up again in the next iteration to complete the testing. So generally, it fits in towards the end of the iteration.
In having said that, getting all your automated testing done in a single iteration isn’t always possible. For example, at the start of a new project we may not have a mature automated testing strategy in place. It’s something we’ll probably discover and figure out as we go. But at a minimum, the unit tests should be there along with whatever else we can get going early in the project.
Deciding which tests to automate (and which not to) is a decision every team faces on their project. Unit tests are cheap and easy – every project should be writing a ton of these. Integration tests are a bit more complex and involve making trade-offs between costs, scalability and fragility. For example a nice end-to-end integration or smoke screen test can be extremely valuable for a team that just wants to know whether the latest push of their software into a development environment is working or not.
Automated GUI tests on the other hand can be extremely fragile and problematic. It’s not that you should never do them, you just have to be careful as you don’t want to waste a lot of teams on fixing a changing UI test because the GUI hasn’t settled down.
2. What type of test automation should we do in agile projects?
At a minimum, teams should be doing a lot of unit testing. These tests developers write to ensure that their software (at the method and class level) works as expected. Every team should be writing many of these.
After that, it really depends on the nature of your project and your team. If your system is primarily back-office, integration tests (tests verifying all your various subsystems) that are hooked up correctly can be extremely valuable. If you have a GUI / web front end, you may want some high-level smoke screen tests using a tool like Selenium or something.
3. What are all the challenges for a test automation team in agile?
There are a couple of challenges with test automation on any team. One is having the courage and discipline to do it. Automated testing is hard work. It means not moving forward and delivering new functionality. It’s hard for some people to stop, make the investment, and automate.
The second challenge of course is knowing what to automate. Teams need to figure and discover on each project which tests are going to pay off and which are going to give a lot of bang for your buck. We know we can’t automated everything (that would be too expensive) but we also know that not automating is going to kill us. So we just need to be flexible, try lots of different tests, keep what works, and drop anything that gets in the way.
4. About test automation script maintenance, should it be considered as a separate user story/ sprint?
No. While it’s OK for teams to reserve 10-20% of an iteration for general bug fixing, refactoring, or test script maintenance, I wouldn’t recommend turning it into a story. Why? Because it’s not something our customers would find very interesting. As much as we can, I prefer reserving stories for pieces of business functionality our customers would get excited about. Testing, refactoring, and general maintenance are generally things they don’t care about.
5. What is the best time to document testing in agile projects, at the beginning from just the user story or at the end after the function is complete and fully understood?
While I like the idea of automating tests at the start of the sprint, I have never been able to pull it off in practice. I think you could do it if you had a really good idea of what was required and a team with the people, skillset, and desire to do it. I just haven’t seen it yet. That would have to be a pretty mature and skilled team.
Also, quite often on new stories we don’t know what the end result is going to look like. It’s often not until the story is nearly complete, and we’ve gone through many iterations with the customer that we (including the customer) truly understand what is required─that makes automating ahead of time hard.
6. If other groups (marketing, development, project management) have cut back on documentation and measurement to become “leaner,” should testers still be required to write test cases, show traceability to user stories, write a test plan and produce a dashboard of metrics on test cases and bug counts?
I don’t see the connection. If one group in the organization is doing something different to become more lean or efficient– good for them. That’s to be applauded. Conversely if other groups aren’t being efficient or lean, that shouldn’t stop us from doing whatever it is we feel we need to do to serve our customers better.
So if writing test cases, tracking traceability, and extensive test plans aren’t adding any value, they should be dropped regardless of what others in the organization are doing.
7. If my team does not do a hardening sprint, when do we run a full regression test suite?
That’s something each team needs to decide for themselves. While I am generally not a fan of hardening sprints, they are sometimes required. The team just takes on a lot of technical debt, or that they start slowing down because they haven’t invested enough in their build and automation tools, so taking a sprint to firm these things up can help.
One obvious time to do a full on regression test is right before any major product release, e.g. a User Acceptance Testing (UAT). That’s something you’ll obviously want to do before going live. Having said that, I believe the goal of every agile project should be to make UAT redundant or a non-event. In other words, the team does such a good job with it’s automated and in house iteration testing, that when UAT comes around, no bugs are found; they’re not there. The team has been building and releasing software of such a high quality that it is no longer required.
I am not saying drop UAT. You’ve got to earn the right to do that. But every day more and more teams are making UAT a non essential activity instead of spending that time and money on more valuable things (improving and adding more features to the existing software).
8. I am on an agile team that does offshore testing. Often, I do not attend the daily stand-up due to time zones. Can agile be successful this way? What are some suggestions to help me be successful in my testing?
I am no expert in offshore testing. Not really my specialty. But I do find it interesting, that whenever projects get into trouble, they bring everyone together and put them in the same room. This is often the single greatest thing you can do to improve the productivity of your team.
Jonathan Rasmusson
