Testing Tools: It Ain’t only About Automation!

Test execution and utility tools that can make your job easier

My first exposure to the necessity for testers to have an array of tools was from the groundbreaking article “Scripts on my Toolbelt” by Danny Faught. Danny laid out the ideal approach to any testing job, and it got me thinking “How can I leverage tools to help me?” I needed a group of tools to get my tasks done. I needed to have a full toolbelt! It makes a lot of sense. A carpenter can’t do everything with a saw, he needs dozens of tools to get the job done.

Typically when people think about testing tools, they immediately think about test automation tools. Clearly, test automation is an important task, but it’s only one task. Most test automation is traditional black box test automation. What that means is that automation lets you run black box tests faster than you can run them manually. I’m not against test automation or saying test automation is bad. What I am saying is we need more than just test automation tools. After all, there’s a lot more to software testing tools than black box test automation.

What do Software Testing Tools Do?

I want tools that accomplish these 3 tasks:

1) Do what I do faster, only faster.

This is where test automation comes in, it merely does what you could do yourself hopefully (if you’re investing in a test automation program) a lot faster and a lot more!

An example of this is using a tool for fault injection. Fault injection, or forced error testing (http://en.wikipedia.org/wiki/Fault_injection ) is an essential test task in every testing project. This can be done so much faster by a tool than any human. You can see a dozen tools listed on the Wikipedia page for fault injection.

Working with HTML? If you know W3C standards, you can read HTML code and verify code compliance, but even better, or if you don’t know the standards, you could feed the code into the W3C Validator, and get the analysis done for you in seconds. http://validator.w3.org/

2) Give me information I typically would not see.

Memory meters and memory leaks detectors give me great information. Memwatch is a memory leak detection tool I often use, and there are many other tools like it. http://www.linkdata.se/sourcecode/memwatch/

3) Do tasks I cannot.

Parsing a log is a good example . Many logs are mammoth in size. Sorting thru, finding only 1 type record, tailing only the final x-number of lines… these tasks are impractical for a human, but utilities can do the sorting, filtering and formatting and then export the output for attachment to a bug report in seconds. http://www.log-expert.de/ and Microsoft’s LogParser 2.2 are tools I often use.

Scope of Testing Tools

Test tool discussions today are very, very different from tool discussions just a few years ago. There are test management and measurement tools, which I won’t even attempt to cover. Even the scope of tools available for use during test execution is vast. There are thousands. I’ve also had to leave out security, performance, load, monitoring, all kinds of developer tools from unit testing harnesses to static and dynamic code coverage and analysis tools…There are just too many different types of tools to do justice to in a single article. My goal is to open your eyes to a broader array of QA testing, even black or gray box testing tools.

The mobile explosion has led to a surge in mobile platform tools and cross-platform tools that are often differentiated by how many testing tools they include. And, there are new testing frontiers many teams did not previously face, like testing download speed, connectivity speed, the number of server requests, amount of data downloads etc.. All of these things impacts what we as mobile users are changed, and they need to be tested. We need tools that will give us that type of information.

ALM tools have mainly been used for task management, project management and measurement, yet a big selling point for administrative tool suites are the test tools integrations. Every ALM suite now has integration of a unit testing harness and continuous integration (CI) tool, and some also integrate with GUI or UI driven test tools, code coverage analysis tools, capture/reply tools, and various others.

There have always been some variety of testing tools packaged with IDEs (Integrated Development Environments) but they almost universally focused on tools for developers, and were very limited in scope. This has changed. Take Visual Studio/TFS for example. In addition to the breakthrough Coded UI tool, it now has an Action recorder, intelliTrace, event log , system info, network emulation, test impact and built-in video recorder, all of which can provide great benefits to testers. Coded UI can be extended from a white-box unit testing tool to a black-box test automation tool by using the TestArchitect for Visual Studio (TA4VS) plug-in, a action-driven test development extension for creating and, maintaining, Coded UI tests without the hassle of coding.

More importantly these days, there is a greater focus on testing tools instead of only developer tools. I would like to think this is due to greater attention to releasing quality product, but may merely by a result of the need for speedy delivery. Higher demands for speed and quality, from faster development practices, faster distribution and deployment and particularly with mobile development, the speed of delivery, instant feedback and cross-platform nature of mobile apps have all made test tools an integral part of any testing practice rather than nice-to-have.

Many people look exclusively for free or open source tools. This is a big mistake. The justification for free or nothing is often that their company will not pay for testing tools. That does not say very much for that company’s commitment to quality. While there are some great free and open source tools, there are some that are well worth the cost. My arsenal of tools is a mix of the best tools I can get, free or not. BTW, tool and utility are often used interchangeably, but there are some technical distinctions.

Utility software is system software designed to help analyze, configure, optimize or maintain a computer. A single piece of utility software is usually called a utility or tool.

“Tool” is sometimes reserved for items that are more robust, such as having a GUI or more functionality.

* Parsons, June Jamrich; Oja, Dan (2013). New Perspectives on Computer Concepts 2014: Comprehensive.

Choosing Tools

Many people have written about the steps to evaluate and procure tools that are right for you, including integration with other tools sets, cost, usability, etc. Cost is often the easy part. Of course, everyone wants full cross-platform support but that’s a big ask. My focus is only to point out key points of selecting the tools to solve your problems.

First, define your tool needs,

  • What problem are you trying to solve?
  • What tasks take too much time?
  • What information do you not have?

After finding a tool to solve your testing problem, you need to evaluate, among other things:

  • Building expertise and training
  • Installation and maintenance
  • The amount of programming skill needed

It is important to remember a great source of testing tools— your development team! They will be familiar with the exact needs, they can perfectly fit the tool to the platform, and more. The biggest problem with your dev team creating tools for testing is lack of time. Most dev teams I know rarely have time to build tools for their test team. But it is worth asking!

Random Tools for Your Toolbelt

I have a prejudice toward testing tools that give me information I can’t get through black box tests. I like tools that give me information at the system level. I like to see what APIs are loaded. I like to find out how much memory is being used when I do certain tasks.

One of my favorite tools is Dependancy walker. (http://www.dependencywalker.com/ ) It is a free utility that builds a hierarchical tree diagram of all dependent modules. For each module found, it lists all the functions that are exported by that module, and which of those functions are actually being called by other modules. To test SQL Injection for databases, definitely find tools! A couple of others I like are Mozilla’s SQL inject me add-in, https://addons.mozilla.org/en-US/firefox/addon/sql-inject-me/ and SQL maphttp://sqlmap.org/. SQL Map has many utilities including penetration testing.

Tips for Finding Tools

Don’t forget logs. You may or may not be aware of all the logs readily available for your reference. From keystrokes to databases and everything in between, every action is logged somewhere.

What platforms or ALM suites are programmers developing your product on? Perl, c#, JavaScript? HTML? SQL? Objective C? every development IDE has lots of tools. Check them out and ask the developers.

See what your team already has. Many times with staff turnover, tools that are a perfect fit get “lost” or stop being used. Also ask other testing teams in your company. For some strange reason, test teams sometimes live in a silo with little contact. They may have tools you can share and visa-versa.

Use in Bug Analysis

Bugs are very often obvious but just as often the causes of bugs can be mysterious. A special use of tools is defect isolation, reproduction and reporting. Many of the tools I’ve listed can assist you in dealing with bugs and reporting excellent bugs in excellent bug reports. Along with recorders, trace analyzers, logs, DependancyWalker , and screenshots bugs do not have to be so mysterious.


My recommendations to you:

  • Look around to find sites that list tools specific to your development platform
  • Look for tools built into your ALM or IDE.
  • Ask your Devs.
  • And, Go hunting— you may be surprised at what you find.

BTW, Monkey has become a very popular name for tools in the mobile arena. A monkey is a cute name for a utility or tool that does a boring, mundane task for you.

Go hunt for some monkeys!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

People who know me and my work probably know my emphasis on good test design for successful test automation. I have written about this in “Key Success Factors for Keyword Driven Testing“. In the Action Based Testing (ABT) method that I have pioneered over the years it is an essential element for success. However, agreeing ...
Are you frustrated with vendors of test automation tools that do not tell you the whole story about what it takes to automate testing? Are you tired of trying to implement test automation without breaking the bank and without overloading yourself with work? I experienced first-hand why people find test automation difficult, and I developed ...
LogiGear Magazine January Trends Issue 2017
A short-list of selection criteria and popular automation tools. There are a lot of test automation tools available in the market, from heavy-duty enterprise level tools to quick and dirty playback-and-record tools for browser testing. For anyone just starting their research we’ve put together a short list of requirements and tools to consider.
When automated tests are well-organized and written with the necessary detail, they can be very efficient and maintainable. But designing automated tests that deal with data can be challenging if you have a lot of data combinations. For example, let’s say we want to simulate a series of 20 customers, along with the number of ...
September Issue 2018: The Secrets to Better Test Automation  
Introduction Many executives have some very basic questions about Software Testing. These questions address the elements of quality (customer satisfaction) and money (spending the least amount of money to prevent future loss). The basic questions that executive have about Software Testing include: Why care about and spend money on testing? Why should testing be treated ...
Identifying which tests to begin with when starting automation is key to driving testing cycle times down and coverage up. So there you are. You’ve done a little research and made the business case to upper management regarding test automation and they bit on the proposal. Surprisingly, they supported you all the way and are extremely ...
How to do UI test automation with the fewest headaches I’m currently interviewing lots of teams that have implemented acceptance testing for my new book. A majority of those interviewed so far have at some point shot themselves in the foot with UI test automation. After speaking to several people who are about to do ...
Investing in Test Automation training will increase your team’s productivity. The availability of reliable jobs in a competitive US market seems to be constantly embattled with competition and replacements of artificial intelligence (AI). In 2016, Foxconn replaced 60,000 employees with robots. However, the growth of Test Automation as an occupation has highlighted an intriguing option ...
Developers of large data-intensive software often notice an interesting — though not surprising — phenomenon: When usage of an application jumps dramatically, components that have operated for months without trouble suddenly develop previously undetected errors. For example, the application may have been installed on a different OS-hardware-DBMS-networking platform, or newly added customers may have account ...
“Happy About Global Software Test Automation: A Discussion of Software Testing for Executives” Author: Hung Q. Nguyen, Michael Hackett, and Brent K. Whitlock Publisher: Happy About (August 1, 2006) Finally, a testing book for executives!, November 17, 2006 By Scott Barber “Chief Technologist, PerfTestPlus” Happy About Global Software Test Automation: A Discussion of Software Testing ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news