Testing Tools: It Ain’t only About Automation!

Test execution and utility tools that can make your job easier

My first exposure to the necessity for testers to have an array of tools was from the groundbreaking article “Scripts on my Toolbelt” by Danny Faught. Danny laid out the ideal approach to any testing job, and it got me thinking “How can I leverage tools to help me?” I needed a group of tools to get my tasks done. I needed to have a full toolbelt! It makes a lot of sense. A carpenter can’t do everything with a saw, he needs dozens of tools to get the job done.

Typically when people think about testing tools, they immediately think about test automation tools. Clearly, test automation is an important task, but it’s only one task. Most test automation is traditional black box test automation. What that means is that automation lets you run black box tests faster than you can run them manually. I’m not against test automation or saying test automation is bad. What I am saying is we need more than just test automation tools. After all, there’s a lot more to software testing tools than black box test automation.

What do Software Testing Tools Do?

I want tools that accomplish these 3 tasks:

1) Do what I do faster, only faster.

This is where test automation comes in, it merely does what you could do yourself hopefully (if you’re investing in a test automation program) a lot faster and a lot more!

An example of this is using a tool for fault injection. Fault injection, or forced error testing (http://en.wikipedia.org/wiki/Fault_injection ) is an essential test task in every testing project. This can be done so much faster by a tool than any human. You can see a dozen tools listed on the Wikipedia page for fault injection.

Working with HTML? If you know W3C standards, you can read HTML code and verify code compliance, but even better, or if you don’t know the standards, you could feed the code into the W3C Validator, and get the analysis done for you in seconds. http://validator.w3.org/

2) Give me information I typically would not see.

Memory meters and memory leaks detectors give me great information. Memwatch is a memory leak detection tool I often use, and there are many other tools like it. http://www.linkdata.se/sourcecode/memwatch/

3) Do tasks I cannot.

Parsing a log is a good example . Many logs are mammoth in size. Sorting thru, finding only 1 type record, tailing only the final x-number of lines… these tasks are impractical for a human, but utilities can do the sorting, filtering and formatting and then export the output for attachment to a bug report in seconds. http://www.log-expert.de/ and Microsoft’s LogParser 2.2 are tools I often use.

Scope of Testing Tools

Test tool discussions today are very, very different from tool discussions just a few years ago. There are test management and measurement tools, which I won’t even attempt to cover. Even the scope of tools available for use during test execution is vast. There are thousands. I’ve also had to leave out security, performance, load, monitoring, all kinds of developer tools from unit testing harnesses to static and dynamic code coverage and analysis tools…There are just too many different types of tools to do justice to in a single article. My goal is to open your eyes to a broader array of QA testing, even black or gray box testing tools.

The mobile explosion has led to a surge in mobile platform tools and cross-platform tools that are often differentiated by how many testing tools they include. And, there are new testing frontiers many teams did not previously face, like testing download speed, connectivity speed, the number of server requests, amount of data downloads etc.. All of these things impacts what we as mobile users are changed, and they need to be tested. We need tools that will give us that type of information.

ALM tools have mainly been used for task management, project management and measurement, yet a big selling point for administrative tool suites are the test tools integrations. Every ALM suite now has integration of a unit testing harness and continuous integration (CI) tool, and some also integrate with GUI or UI driven test tools, code coverage analysis tools, capture/reply tools, and various others.

There have always been some variety of testing tools packaged with IDEs (Integrated Development Environments) but they almost universally focused on tools for developers, and were very limited in scope. This has changed. Take Visual Studio/TFS for example. In addition to the breakthrough Coded UI tool, it now has an Action recorder, intelliTrace, event log , system info, network emulation, test impact and built-in video recorder, all of which can provide great benefits to testers. Coded UI can be extended from a white-box unit testing tool to a black-box test automation tool by using the TestArchitect for Visual Studio (TA4VS) plug-in, a action-driven test development extension for creating and, maintaining, Coded UI tests without the hassle of coding.

More importantly these days, there is a greater focus on testing tools instead of only developer tools. I would like to think this is due to greater attention to releasing quality product, but may merely by a result of the need for speedy delivery. Higher demands for speed and quality, from faster development practices, faster distribution and deployment and particularly with mobile development, the speed of delivery, instant feedback and cross-platform nature of mobile apps have all made test tools an integral part of any testing practice rather than nice-to-have.

Many people look exclusively for free or open source tools. This is a big mistake. The justification for free or nothing is often that their company will not pay for testing tools. That does not say very much for that company’s commitment to quality. While there are some great free and open source tools, there are some that are well worth the cost. My arsenal of tools is a mix of the best tools I can get, free or not. BTW, tool and utility are often used interchangeably, but there are some technical distinctions.

Utility software is system software designed to help analyze, configure, optimize or maintain a computer. A single piece of utility software is usually called a utility or tool.

“Tool” is sometimes reserved for items that are more robust, such as having a GUI or more functionality.

* Parsons, June Jamrich; Oja, Dan (2013). New Perspectives on Computer Concepts 2014: Comprehensive.

Choosing Tools

Many people have written about the steps to evaluate and procure tools that are right for you, including integration with other tools sets, cost, usability, etc. Cost is often the easy part. Of course, everyone wants full cross-platform support but that’s a big ask. My focus is only to point out key points of selecting the tools to solve your problems.

First, define your tool needs,

  • What problem are you trying to solve?
  • What tasks take too much time?
  • What information do you not have?

After finding a tool to solve your testing problem, you need to evaluate, among other things:

  • Building expertise and training
  • Installation and maintenance
  • The amount of programming skill needed

It is important to remember a great source of testing tools— your development team! They will be familiar with the exact needs, they can perfectly fit the tool to the platform, and more. The biggest problem with your dev team creating tools for testing is lack of time. Most dev teams I know rarely have time to build tools for their test team. But it is worth asking!

Random Tools for Your Toolbelt

I have a prejudice toward testing tools that give me information I can’t get through black box tests. I like tools that give me information at the system level. I like to see what APIs are loaded. I like to find out how much memory is being used when I do certain tasks.

One of my favorite tools is Dependancy walker. (http://www.dependencywalker.com/ ) It is a free utility that builds a hierarchical tree diagram of all dependent modules. For each module found, it lists all the functions that are exported by that module, and which of those functions are actually being called by other modules. To test SQL Injection for databases, definitely find tools! A couple of others I like are Mozilla’s SQL inject me add-in, https://addons.mozilla.org/en-US/firefox/addon/sql-inject-me/ and SQL maphttp://sqlmap.org/. SQL Map has many utilities including penetration testing.

Tips for Finding Tools

Don’t forget logs. You may or may not be aware of all the logs readily available for your reference. From keystrokes to databases and everything in between, every action is logged somewhere.

What platforms or ALM suites are programmers developing your product on? Perl, c#, JavaScript? HTML? SQL? Objective C? every development IDE has lots of tools. Check them out and ask the developers.

See what your team already has. Many times with staff turnover, tools that are a perfect fit get “lost” or stop being used. Also ask other testing teams in your company. For some strange reason, test teams sometimes live in a silo with little contact. They may have tools you can share and visa-versa.

Use in Bug Analysis

Bugs are very often obvious but just as often the causes of bugs can be mysterious. A special use of tools is defect isolation, reproduction and reporting. Many of the tools I’ve listed can assist you in dealing with bugs and reporting excellent bugs in excellent bug reports. Along with recorders, trace analyzers, logs, DependancyWalker , and screenshots bugs do not have to be so mysterious.


My recommendations to you:

  • Look around to find sites that list tools specific to your development platform
  • Look for tools built into your ALM or IDE.
  • Ask your Devs.
  • And, Go hunting— you may be surprised at what you find.

BTW, Monkey has become a very popular name for tools in the mobile arena. A monkey is a cute name for a utility or tool that does a boring, mundane task for you.

Go hunt for some monkeys!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

As our world continues its digital transformation with excitement in the advancement and convergence of so many technologies- from AI, machine learning, big data and analytics, to device mesh connectivity, nor should we forget VR and AR- 2017 promises to be a year that further transforms the way we work, play and take care of ...
LogiGear Magazine – April 2013 – Test Automation
In order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.
June Issue 2019: Testing the Software Car
When automated tests are well-organized and written with the necessary detail, they can be very efficient and maintainable. But designing automated tests that deal with data can be challenging if you have a lot of data combinations. For example, let’s say we want to simulate a series of 20 customers, along with the number of ...
I recently came back from the Software Testing & Evaluation Summit in Washington, DC hosted by the National Defense Industrial Association. The objective of the workshop is to help recommend policy and guidance changes to the Defense enterprise, focusing on improving practice and productivity of software testing and evaluation (T&E) approaches in Defense acquisition.
This book isn’t for everyone, but everyone can get some value out of it. What I mean by that rather confusing statement is that folks working in Agile environments will likely want to throw the book across the room while folks in more bureaucratic environments like CMMI or other waterfall environments will likely get a ...
All too often, software development organizations look at automating software testing as a means of executing existing test cases faster. Very frequently there is no strategic or methodological underpinning to such an effort. The approach is one of running test cases faster is better, which will help to deliver software faster. Even in organizations that ...
LogiGear Magazine September Issue 2020: Testing Transformations: Modernizing QA in the SDLC
Understanding the benefits and challenges of Automating ERP is critical. According to SAP, ERP (Enterprise Resource Planning) “is the core processes that are needed to run a company: finance, human resources, manufacturing, supply chain, services, procurement, and others. At its most basic level, ERP integrates these processes into a single system. But new ERP systems ...
Test Automation is significant and growing-yet I have read many forum comments and blog posts about Test Automation not delivering as expected. It’s true that test automation can improve reliability while minimizing variability in the results, speed up the process, increase test coverage, and ultimately provide greater confidence in the quality of the software being ...
Over the years, we’ve provided an extensive number of articles that provide a wealth of knowledge about Test Automation. Below is a compilation of some of those articles. Guide to Automated Voice Apps Testing This article explores some of the basic test tools you’ll need and how to blend the use different automated testing tools ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news