Testing Tools: It Ain’t only About Automation!

Test execution and utility tools that can make your job easier

My first exposure to the necessity for testers to have an array of tools was from the groundbreaking article “Scripts on my Toolbelt” by Danny Faught. Danny laid out the ideal approach to any testing job, and it got me thinking “How can I leverage tools to help me?” I needed a group of tools to get my tasks done. I needed to have a full toolbelt! It makes a lot of sense. A carpenter can’t do everything with a saw, he needs dozens of tools to get the job done.

Typically when people think about testing tools, they immediately think about test automation tools. Clearly, test automation is an important task, but it’s only one task. Most test automation is traditional black box test automation. What that means is that automation lets you run black box tests faster than you can run them manually. I’m not against test automation or saying test automation is bad. What I am saying is we need more than just test automation tools. After all, there’s a lot more to software testing tools than black box test automation.

What do Software Testing Tools Do?

I want tools that accomplish these 3 tasks:

1) Do what I do faster, only faster.

This is where test automation comes in, it merely does what you could do yourself hopefully (if you’re investing in a test automation program) a lot faster and a lot more!

An example of this is using a tool for fault injection. Fault injection, or forced error testing (http://en.wikipedia.org/wiki/Fault_injection ) is an essential test task in every testing project. This can be done so much faster by a tool than any human. You can see a dozen tools listed on the Wikipedia page for fault injection.

Working with HTML? If you know W3C standards, you can read HTML code and verify code compliance, but even better, or if you don’t know the standards, you could feed the code into the W3C Validator, and get the analysis done for you in seconds. http://validator.w3.org/

2) Give me information I typically would not see.

Memory meters and memory leaks detectors give me great information. Memwatch is a memory leak detection tool I often use, and there are many other tools like it. http://www.linkdata.se/sourcecode/memwatch/

3) Do tasks I cannot.

Parsing a log is a good example . Many logs are mammoth in size. Sorting thru, finding only 1 type record, tailing only the final x-number of lines… these tasks are impractical for a human, but utilities can do the sorting, filtering and formatting and then export the output for attachment to a bug report in seconds. http://www.log-expert.de/ and Microsoft’s LogParser 2.2 are tools I often use.

Scope of Testing Tools

Test tool discussions today are very, very different from tool discussions just a few years ago. There are test management and measurement tools, which I won’t even attempt to cover. Even the scope of tools available for use during test execution is vast. There are thousands. I’ve also had to leave out security, performance, load, monitoring, all kinds of developer tools from unit testing harnesses to static and dynamic code coverage and analysis tools…There are just too many different types of tools to do justice to in a single article. My goal is to open your eyes to a broader array of QA testing, even black or gray box testing tools.

The mobile explosion has led to a surge in mobile platform tools and cross-platform tools that are often differentiated by how many testing tools they include. And, there are new testing frontiers many teams did not previously face, like testing download speed, connectivity speed, the number of server requests, amount of data downloads etc.. All of these things impacts what we as mobile users are changed, and they need to be tested. We need tools that will give us that type of information.

ALM tools have mainly been used for task management, project management and measurement, yet a big selling point for administrative tool suites are the test tools integrations. Every ALM suite now has integration of a unit testing harness and continuous integration (CI) tool, and some also integrate with GUI or UI driven test tools, code coverage analysis tools, capture/reply tools, and various others.

There have always been some variety of testing tools packaged with IDEs (Integrated Development Environments) but they almost universally focused on tools for developers, and were very limited in scope. This has changed. Take Visual Studio/TFS for example. In addition to the breakthrough Coded UI tool, it now has an Action recorder, intelliTrace, event log , system info, network emulation, test impact and built-in video recorder, all of which can provide great benefits to testers. Coded UI can be extended from a white-box unit testing tool to a black-box test automation tool by using the TestArchitect for Visual Studio (TA4VS) plug-in, a action-driven test development extension for creating and, maintaining, Coded UI tests without the hassle of coding.

More importantly these days, there is a greater focus on testing tools instead of only developer tools. I would like to think this is due to greater attention to releasing quality product, but may merely by a result of the need for speedy delivery. Higher demands for speed and quality, from faster development practices, faster distribution and deployment and particularly with mobile development, the speed of delivery, instant feedback and cross-platform nature of mobile apps have all made test tools an integral part of any testing practice rather than nice-to-have.

Many people look exclusively for free or open source tools. This is a big mistake. The justification for free or nothing is often that their company will not pay for testing tools. That does not say very much for that company’s commitment to quality. While there are some great free and open source tools, there are some that are well worth the cost. My arsenal of tools is a mix of the best tools I can get, free or not. BTW, tool and utility are often used interchangeably, but there are some technical distinctions.

Utility software is system software designed to help analyze, configure, optimize or maintain a computer. A single piece of utility software is usually called a utility or tool.

“Tool” is sometimes reserved for items that are more robust, such as having a GUI or more functionality.

* Parsons, June Jamrich; Oja, Dan (2013). New Perspectives on Computer Concepts 2014: Comprehensive.

Choosing Tools

Many people have written about the steps to evaluate and procure tools that are right for you, including integration with other tools sets, cost, usability, etc. Cost is often the easy part. Of course, everyone wants full cross-platform support but that’s a big ask. My focus is only to point out key points of selecting the tools to solve your problems.

First, define your tool needs,

  • What problem are you trying to solve?
  • What tasks take too much time?
  • What information do you not have?

After finding a tool to solve your testing problem, you need to evaluate, among other things:

  • Building expertise and training
  • Installation and maintenance
  • The amount of programming skill needed

It is important to remember a great source of testing tools— your development team! They will be familiar with the exact needs, they can perfectly fit the tool to the platform, and more. The biggest problem with your dev team creating tools for testing is lack of time. Most dev teams I know rarely have time to build tools for their test team. But it is worth asking!

Random Tools for Your Toolbelt

I have a prejudice toward testing tools that give me information I can’t get through black box tests. I like tools that give me information at the system level. I like to see what APIs are loaded. I like to find out how much memory is being used when I do certain tasks.

One of my favorite tools is Dependancy walker. (http://www.dependencywalker.com/ ) It is a free utility that builds a hierarchical tree diagram of all dependent modules. For each module found, it lists all the functions that are exported by that module, and which of those functions are actually being called by other modules. To test SQL Injection for databases, definitely find tools! A couple of others I like are Mozilla’s SQL inject me add-in, https://addons.mozilla.org/en-US/firefox/addon/sql-inject-me/ and SQL maphttp://sqlmap.org/. SQL Map has many utilities including penetration testing.

Tips for Finding Tools

Don’t forget logs. You may or may not be aware of all the logs readily available for your reference. From keystrokes to databases and everything in between, every action is logged somewhere.

What platforms or ALM suites are programmers developing your product on? Perl, c#, JavaScript? HTML? SQL? Objective C? every development IDE has lots of tools. Check them out and ask the developers.

See what your team already has. Many times with staff turnover, tools that are a perfect fit get “lost” or stop being used. Also ask other testing teams in your company. For some strange reason, test teams sometimes live in a silo with little contact. They may have tools you can share and visa-versa.

Use in Bug Analysis

Bugs are very often obvious but just as often the causes of bugs can be mysterious. A special use of tools is defect isolation, reproduction and reporting. Many of the tools I’ve listed can assist you in dealing with bugs and reporting excellent bugs in excellent bug reports. Along with recorders, trace analyzers, logs, DependancyWalker , and screenshots bugs do not have to be so mysterious.


My recommendations to you:

  • Look around to find sites that list tools specific to your development platform
  • Look for tools built into your ALM or IDE.
  • Ask your Devs.
  • And, Go hunting— you may be surprised at what you find.

BTW, Monkey has become a very popular name for tools in the mobile arena. A monkey is a cute name for a utility or tool that does a boring, mundane task for you.

Go hunt for some monkeys!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

Framework: An abstraction in which software providing generic functionality can be selectively changed by additional user written code, thus providing application specific software. A software framework is a universal, reusable software platform used to develop applications, products and solutions. Harness: A collection of software and test data configured to test a program unit by running it under varying conditions and monitoring ...
The huge range of mobile devices used to browse the web now means testing a mobile website before delivery is critical.
There are few topics in quality assurance testing that cause as much confusion as smoke testing versus sanity testing. The two names would seem to describe very different practices— and they do! But people still get them confused, since the distinction is somewhat subtle.
Mobile usage today is not just a trend but it is an essential shift in how people communicate with each other, interact with the world, and do business. According to a ComScore, in 2014 the number of mobile users surpassed the number of computer users and is showing strong growth over time, towards some point in ...
Are you frustrated with vendors of test automation tools that do not tell you the whole story about what it takes to automate testing? Are you tired of trying to implement test automation without breaking the bank and without overloading yourself with work? I experienced first-hand why people find test automation difficult, and I developed ...
I feel like I’ve spent most of my career learning how to write good automated tests in an Agile environment. When I downloaded JUnit in the year 2000 it didn’t take long before I was hooked – unit tests for everything in sight. That gratifying green bar is near-instant feedback that everything is going as ...
How lagging automotive design principles adversely affect final products. Cars are integrating more and more software with every model year. The ginormous screen introduced by Tesla in their flagship Model S a few years ago was seemingly unrivaled at the time. Nowadays, screens of this size are not only commonplace in vehicles such as the ...
LogiGear Magazine September Test Automation Issue 2017
Introduction In many of the Test Automation projects that we are involved with using our Action-Based Testing methodology, management has expressed a need to relate tests and test results to system requirements. The underlying thought is that automation will create extra possibilities to control the level of compliance to requirements of the system under test. ...
In order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.
“Testing Applications on the web” – 2nd EditionAuthors: Hung Q. Nguyen, Bob Johnson, Michael HackettPublisher: Wiley; edition (May 16, 2003) This is good book. If you test web apps, you should buy it!, April 20, 2001By Dr. Cem Kaner – Director of Florida Institute of Technology’s Center for Software Testing Education & Research Book Reviews ...
The guide for CUI Automated Testing strategies, including chatbot testing and voice app testing. In the Software Testing industry, trends come and go that shape the future of testing. From Automation in Agile, to the DevOps era we are now in, trends are what evolve and improve our testing processes and ideologies. Currently, many researchers ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news