Testing Tools: It Ain’t only About Automation!

Test execution and utility tools that can make your job easier

My first exposure to the necessity for testers to have an array of tools was from the groundbreaking article “Scripts on my Toolbelt” by Danny Faught. Danny laid out the ideal approach to any testing job, and it got me thinking “How can I leverage tools to help me?” I needed a group of tools to get my tasks done. I needed to have a full toolbelt! It makes a lot of sense. A carpenter can’t do everything with a saw, he needs dozens of tools to get the job done.

Typically when people think about testing tools, they immediately think about test automation tools. Clearly, test automation is an important task, but it’s only one task. Most test automation is traditional black box test automation. What that means is that automation lets you run black box tests faster than you can run them manually. I’m not against test automation or saying test automation is bad. What I am saying is we need more than just test automation tools. After all, there’s a lot more to software testing tools than black box test automation.

What do Software Testing Tools Do?

I want tools that accomplish these 3 tasks:

1) Do what I do faster, only faster.

This is where test automation comes in, it merely does what you could do yourself hopefully (if you’re investing in a test automation program) a lot faster and a lot more!

An example of this is using a tool for fault injection. Fault injection, or forced error testing (http://en.wikipedia.org/wiki/Fault_injection ) is an essential test task in every testing project. This can be done so much faster by a tool than any human. You can see a dozen tools listed on the Wikipedia page for fault injection.

Working with HTML? If you know W3C standards, you can read HTML code and verify code compliance, but even better, or if you don’t know the standards, you could feed the code into the W3C Validator, and get the analysis done for you in seconds. http://validator.w3.org/

2) Give me information I typically would not see.

Memory meters and memory leaks detectors give me great information. Memwatch is a memory leak detection tool I often use, and there are many other tools like it. http://www.linkdata.se/sourcecode/memwatch/

3) Do tasks I cannot.

Parsing a log is a good example . Many logs are mammoth in size. Sorting thru, finding only 1 type record, tailing only the final x-number of lines… these tasks are impractical for a human, but utilities can do the sorting, filtering and formatting and then export the output for attachment to a bug report in seconds. http://www.log-expert.de/ and Microsoft’s LogParser 2.2 are tools I often use.

Scope of Testing Tools

Test tool discussions today are very, very different from tool discussions just a few years ago. There are test management and measurement tools, which I won’t even attempt to cover. Even the scope of tools available for use during test execution is vast. There are thousands. I’ve also had to leave out security, performance, load, monitoring, all kinds of developer tools from unit testing harnesses to static and dynamic code coverage and analysis tools…There are just too many different types of tools to do justice to in a single article. My goal is to open your eyes to a broader array of QA testing, even black or gray box testing tools.

The mobile explosion has led to a surge in mobile platform tools and cross-platform tools that are often differentiated by how many testing tools they include. And, there are new testing frontiers many teams did not previously face, like testing download speed, connectivity speed, the number of server requests, amount of data downloads etc.. All of these things impacts what we as mobile users are changed, and they need to be tested. We need tools that will give us that type of information.

ALM tools have mainly been used for task management, project management and measurement, yet a big selling point for administrative tool suites are the test tools integrations. Every ALM suite now has integration of a unit testing harness and continuous integration (CI) tool, and some also integrate with GUI or UI driven test tools, code coverage analysis tools, capture/reply tools, and various others.

There have always been some variety of testing tools packaged with IDEs (Integrated Development Environments) but they almost universally focused on tools for developers, and were very limited in scope. This has changed. Take Visual Studio/TFS for example. In addition to the breakthrough Coded UI tool, it now has an Action recorder, intelliTrace, event log , system info, network emulation, test impact and built-in video recorder, all of which can provide great benefits to testers. Coded UI can be extended from a white-box unit testing tool to a black-box test automation tool by using the TestArchitect for Visual Studio (TA4VS) plug-in, a action-driven test development extension for creating and, maintaining, Coded UI tests without the hassle of coding.

More importantly these days, there is a greater focus on testing tools instead of only developer tools. I would like to think this is due to greater attention to releasing quality product, but may merely by a result of the need for speedy delivery. Higher demands for speed and quality, from faster development practices, faster distribution and deployment and particularly with mobile development, the speed of delivery, instant feedback and cross-platform nature of mobile apps have all made test tools an integral part of any testing practice rather than nice-to-have.

Many people look exclusively for free or open source tools. This is a big mistake. The justification for free or nothing is often that their company will not pay for testing tools. That does not say very much for that company’s commitment to quality. While there are some great free and open source tools, there are some that are well worth the cost. My arsenal of tools is a mix of the best tools I can get, free or not. BTW, tool and utility are often used interchangeably, but there are some technical distinctions.

Utility software is system software designed to help analyze, configure, optimize or maintain a computer. A single piece of utility software is usually called a utility or tool.

“Tool” is sometimes reserved for items that are more robust, such as having a GUI or more functionality.

* Parsons, June Jamrich; Oja, Dan (2013). New Perspectives on Computer Concepts 2014: Comprehensive.

Choosing Tools

Many people have written about the steps to evaluate and procure tools that are right for you, including integration with other tools sets, cost, usability, etc. Cost is often the easy part. Of course, everyone wants full cross-platform support but that’s a big ask. My focus is only to point out key points of selecting the tools to solve your problems.

First, define your tool needs,

  • What problem are you trying to solve?
  • What tasks take too much time?
  • What information do you not have?

After finding a tool to solve your testing problem, you need to evaluate, among other things:

  • Building expertise and training
  • Installation and maintenance
  • The amount of programming skill needed

It is important to remember a great source of testing tools— your development team! They will be familiar with the exact needs, they can perfectly fit the tool to the platform, and more. The biggest problem with your dev team creating tools for testing is lack of time. Most dev teams I know rarely have time to build tools for their test team. But it is worth asking!

Random Tools for Your Toolbelt

I have a prejudice toward testing tools that give me information I can’t get through black box tests. I like tools that give me information at the system level. I like to see what APIs are loaded. I like to find out how much memory is being used when I do certain tasks.

One of my favorite tools is Dependancy walker. (http://www.dependencywalker.com/ ) It is a free utility that builds a hierarchical tree diagram of all dependent modules. For each module found, it lists all the functions that are exported by that module, and which of those functions are actually being called by other modules. To test SQL Injection for databases, definitely find tools! A couple of others I like are Mozilla’s SQL inject me add-in, https://addons.mozilla.org/en-US/firefox/addon/sql-inject-me/ and SQL maphttp://sqlmap.org/. SQL Map has many utilities including penetration testing.

Tips for Finding Tools

Don’t forget logs. You may or may not be aware of all the logs readily available for your reference. From keystrokes to databases and everything in between, every action is logged somewhere.

What platforms or ALM suites are programmers developing your product on? Perl, c#, JavaScript? HTML? SQL? Objective C? every development IDE has lots of tools. Check them out and ask the developers.

See what your team already has. Many times with staff turnover, tools that are a perfect fit get “lost” or stop being used. Also ask other testing teams in your company. For some strange reason, test teams sometimes live in a silo with little contact. They may have tools you can share and visa-versa.

Use in Bug Analysis

Bugs are very often obvious but just as often the causes of bugs can be mysterious. A special use of tools is defect isolation, reproduction and reporting. Many of the tools I’ve listed can assist you in dealing with bugs and reporting excellent bugs in excellent bug reports. Along with recorders, trace analyzers, logs, DependancyWalker , and screenshots bugs do not have to be so mysterious.


My recommendations to you:

  • Look around to find sites that list tools specific to your development platform
  • Look for tools built into your ALM or IDE.
  • Ask your Devs.
  • And, Go hunting— you may be surprised at what you find.

BTW, Monkey has become a very popular name for tools in the mobile arena. A monkey is a cute name for a utility or tool that does a boring, mundane task for you.

Go hunt for some monkeys!

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

LogiGear Magazine – April 2014 – Test Tool and Automation
Introduction A characteristic of data warehouse (DW) development is the frequent release of high-quality data for user feedback and acceptance. At the end of each iteration of DW ETLs (Extract-Transform-Load), data tables are expected to be of sufficient quality for the next ETL phase. This objective requires a unique approach to quality assurance methods and ...
When Netflix decided to enter the Android ecosystem, we faced a daunting set of challenges: 1. We wanted to release rapidly (every 6-8 weeks). 2. There were hundreds of Android devices of different shapes, versions, capacities, and specifications which need to playback audio and video. 3. We wanted to keep the team small and happy. ...
This article was developed from concepts in the book Global Software Test Automation: A Discussion of Software Testing for Executives, by Hung Q. Nguyen, Michael Hacket and Brent K. Whitlock Introduction The top 5 pitfalls encountered by managers employing software Test Automation are: Uncertainty and lack of control Poor scalability and maintainability Low Test Automation ...
Understanding the benefits and challenges of Automating ERP is critical. According to SAP, ERP (Enterprise Resource Planning) “is the core processes that are needed to run a company: finance, human resources, manufacturing, supply chain, services, procurement, and others. At its most basic level, ERP integrates these processes into a single system. But new ERP systems ...
Based in Alberta, Canada, Jonathan Kohl takes time out of his busy schedule to discuss his views on software testing and automation.
Two dominant manual testing approaches to the software testing game are scripted and exploratory testing. In the test automation space, we have other approaches. I look at three main contexts for test automation: 1. Code context – e.g. unit testing. 2. System context – e.g. protocol or message level testing. 3. Social context – e.g. ...
The Cloud demands that we be as nimble as possible, delivering features and fixes in almost real-time fashion. Both customer and provider rely on software development that can maintain quality while being light on its feet and constantly moving. In addition, Cloud-oriented systems tend to be highly complex and dynamic in structure — more than ...
Many organizations rely on HP Quality Center to design test plans and track test results. TestArchitect’s Quality Center integration makes working with QC as easy as pie. TestArchitect (TA) is a three-in-one tool for Test Management, Test Development, and Test Automation. Users can create and manage test assets, execute tests, track and analyze test results, ...
5 roadblocks in vehicular autonomy that complicate Software Testing Experts in the field have previously referred to air travel as somewhat of a gold standard for autonomous vehicle safety, but after Boeing’s two tragedies, that analogy can no longer be used when talking about self-driving cars. This was after Boeing’s 737 MAX Jets have found ...
The huge range of mobile devices used to browse the web now means testing a mobile website before delivery is critical.
One of my current responsibilities is to find ways to automate, as much as practical, the ‘testing’ of the user experience (UX) for complex web-based applications. In my view, full test automation of UX is impractical and probably unwise; however, we can use automation to find potential UX problems, or undesirable effects, even in rich, ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news