Conversational User Interfaces: Breaking Down the Best Automated Testing Strategies

The guide for CUI Automated Testing strategies, including chatbot testing and voice app testing.

In the Software Testing industry, trends come and go that shape the future of testing. From Automation in Agile, to the DevOps era we are now in, trends are what evolve and improve our testing processes and ideologies. Currently, many researchers believe the next big thing in the testing industry will be the use of artificial intelligence (AI) in conversational user interfaces. In fact, a 2017 study by Forrester forecasted that 25% of jobs will be impacted by AI technologies (including intelligent chatbots) by 2019—a frightening premonition for some, but an open door for those workers who are currently intrigued by the notion of skilling up. But before we dive into that, we need to first understand what a conversational UI is.

What is a Conversational UI?

A conversational user interface (CUI) allows for users to interact with the computer and/or software in a human-like manner. It has evolved from the past versions of communication between humans and computers, where pre-defined scripts were used to respond to customer questions via an instant messenger-esque setup. Nowadays, these UIs mimic conversations one would have with an actual human being. There are 2 overarching types of conversational UIs: (1) a voice assistant that allows you to talk (i.e. Amazon’s Alexa, Google’s “OK Google,” Microsoft’s Cortana, or Apple’s Siri) and (2) chatbots that allow you to type.

Types of Conversational UIs

Voice Assistants

These types of conversational UIs complete actions by having users simply speak a command. Apple was one of the first companies to have their voice assistant, Siri, widely adopted; launched in October 2011, Siri provided iPhone users the ability to access information and complete actions on their device by simply saying, “Hey Siri.” Voice assistants have only been developed further since then; we now have home devices, such as Google Home and Amazon Echo, that allow users to not only access copious amounts of information but also control aspects of their home, including turning the lights on and off, locking and unlocking doors and windows, even purchasing items from the Amazon marketplace simply by using their voice!


Chatbots are mobile or web-based interfaces that allow the user to ask questions and retrieve information. They conduct a conversation via auditory or textual methods. Typically, chatbots are used in dialog systems—a computer system intended to converse with a human—for a variety of tasks, including customer service or acquiring information.

How do CUIs Work?

Many times, conversational interfaces use natural language processing (NLP), a division of artificial intelligence that helps computers to understand and interpret human language. Although natural language processing takes into account the structure of human language (words make a phrase, phrases have meanings, etc.), the ambiguity of human language sometimes baffles NLP. This has led to a shift towards natural language understanding (NLU). NLU attempts to aid in machine reading comprehension by allowing a line of questioning to continue throughout the conversation; it allows the machine to handle unstructured inputs that a human brain could successfully interpret (i.e. spelling errors, follow-up questions, etc.), thus streamlining the entire process of CUIs. While many chatbots use NLP and NLU systems, many others simply scan the inputted query for keywords, and then pull a reply from a database that has the most matching keywords.

Figure 1 –  Example of the TA chatbot process

For example, at LogiGear, we have a chatbot on our TestArchitect Support website. From the backend, we trained our bot by letting it learn our content from the TestArchitect Online Help Knowledge Base. Developers could see the chat log between the bot and the user and would keyword map any related article with the conversation for the bot to learn. By doing this, the next time a user asks a question with the same keywords, the bot can suggest the article on its own accord. You can refer to Figure 1 for a visual depiction of how this works.

Chatbot Testing Strategies

Chatbot testing falls under the denomination of UI/UX testing, which is on the highest level of the testing hierarchy; this is because when you are testing a chatbot, you are testing essentially the entire system and all of the dependent aspects of the system. Despite it being a high level test, the UI of a chatbot is relatively simple: it’s either spoken or written human language. Thus, a test case here would be a conversation that the chatbot should be able to follow. As far as a test suite for chatbots, this usually consists of a library of conversation transcripts. When creating the test suite, you’ll want to identify what Chatbots Magazine describes as the “Happy Paths,”  where the user behaves as expected and continue with the other conversation flows. This way, whenever one of the conversations fails, you will easily be able to identify the error.

Running the test suite manually is quite easy—start Facebook Messenger, open the chatbot, and start chatting! However, this is only feasible for small projects as it can quickly become time consuming and redundant. That is where Test Automation comes in handy. With Automation, you can run your chatbot conversations without you actually driving the conversation; instead, it is essentially a chatbot interacting with another chatbot. All of the tedious work will be completed by the bots, and all you have to do is wait to receive the test reports.

But, this approach only checks to see if the chatbot is continuing the conversation, not if the chatbot actually fulfills the requirements. The approach mentioned above is a verification approach (as opposed to a validation approach); essentially, this method of Automation is checking if the chatbot is actually working as designed, and we assume that it will do the right thing.

Why Test Chatbots?

A quick look into some comments and reviews on early Amazon Alexa products will answer this question for you. From “it asks me if I’m ready but won’t proceed,” to “it doesn’t acknowledge my correct answers!”—it’s pretty evident why we need to test CUIs. One of the biggest challenges in CUIs is the fact that there are no barriers for users; this means that users, either with chatbots or voice activated systems, have absolute freedom to say whatever they wish—meaning CUIs have to cover all kinds of unexpected user inputs. Here’s an example of what I mean:

User: CUI, should I bring a jacket today?

CUI: Where are you going today?

User: Chocolate ice cream!

CUI: …

In this scenario, we’d want the bot to either (a) redirect the conversation or (b) ask the question again. It’s important to design your chatbot test cases with the expectation of unexpected user input. It’s also important to understand a 100% testing coverage rate for CUIs is impossible; because the nature of human language and all of the infinitely possible inputs, we as testers need to plan to cover the most common situations. For instance, having your chatbot know the conversion of cups to liters seems reasonable, but knowing the lifespan of the orchid mantis insect may be unnecessary. We at LogiGear are quite familiar with testing and automating voice apps, including Amazon’s Alexa. We have an entire article about the ins and outs of testing voice assistants, as well as a step-by-step guide and a video. Written by Do Nguyen, you can check it out here.

Importance of CUIs

For Businesses

Chatbots are becoming increasingly prevalent in businesses, as they automate tasks that do not require skill-based talents. There are a plethora of reasons for the implementation of chatbots, one of the most prevalent being efficiency. It’s no secret that software is faster and less prone to errors than humans when it comes to completing tasks. Moreover, these bots can handle astronomically more demands and pieces of information at once than humans.

Time and cost efficiency are definitely beneficial for businesses, but an even larger benefit is the customer service improvement. One study found 53% of customers that ask a question on Twitter expect a response within the hour; having an automated chatbot that works alongside and supports the support team by answering questions 24/7 not only pleases the customers—but also alleviates the pressure these teams may feel. More importantly, it cuts down waiting time for customers with non-specific issues and allows those who actually need the support of a technical support person to experience shorter wait times. Econsultancy reports that 45% of financial service firms are focused on making the customer experience as relevant and precise as possible—and Automation can help with that.

For Software Testers

It’s no secret that chatbots are going to kill jobs; as stated previously in 2017, Forrester forecasted that 25% of jobs will be impacted by AI technologies. But, similar to the first law of thermodynamics—that states that energy can neither be created nor destroyed but rather transformed into a different form—the unskilled labor jobs that are being overtaken are instead being replaced with new jobs and possibilities. A study by Modis found that the tech industry will grow by at least 12% by 2024, and a LinkedIn survey from 2017 confirmed this, finding that business and activities in the tech industry have a growing need for talent. We have an entire article dedicated to this topic, titled Hack Your Test Team’s Productivity, so be sure to check that out!

For the Industry

It is widely known that AI is becoming an increasingly hot topic for all industries, but none more important than the tech industry—this is the industry that builds AI! In fact, Artificial Solutions, one of the industry leaders in enterprise-strength conversational AI, won 4 Stevie 2019 International Business Awards—one of the world’s most coveted prizes for businesses. They took the silver Stevie in the very prestigious Most Innovative Tech Company of the Year, which recognizes a company’s achievement in product innovation. Their software, Teneo, has truly raised the bar for competition conversational AI platforms. Microsoft is one of these companies trying to keep themselves at the forefront of AI innovation; they recently released an updated version of their digital assistant, Cortana, which boasts a new chat-based UI. As opposed to single queries, the new version will show the entire extent of the interaction with Cortana as a means of making the interaction more conversational and more seamless.

One of the most thrilling aspects of what conversational UI means for the Software Testing industry is the unknown aspect of what is to come. It’s obvious that new jobs are going to be created for tech developers that are both ready and willing to work alongside these new technologies. The intelligent virtual assistant (IVA) market—a related field to conversational UIs—is expected to reach a value of $25.63 billion USD by 2025, with a compound annual growth rate of 40.4%. Despite these huge figures, the market is actually expected to grow aggressively over forecasts.


The growth of conversational UIs is both exciting yet worrisome: new technologies are going to drastically improve efficiency in the workplace, improve customer satisfaction, and reduce error—but at the expense of jobs. Yet, we as a global society have already faced a similar workplace situation with the Industrial Revolution; laborers at the time were terrified of what was to come. But, with open minds and growing education, people found ways to work alongside new technology. They say history repeats itself, so hopefully humanity reacts in the same way: uncovering new jobs and industries to work alongside technology. So long as we can do so, conversational UIs will continue to develop and improve business processes worldwide.

Noah Peters
Noah Peters is from the Bay Area. He holds a Bachelor's Degree in Marketing from the University of San Francisco. Noah started at LogiGear as a Marketing Intern and transitioned to a full-time Marketing Associate role post-graduation. Noah is passionate about content creation and SEO, and works closely with all content produced by LogiGear, including the LogiGear Magazine, the LogiGear Blog, and various LogiGear eBooks. In his free time, you can find Noah researching the automotive industry or teaching high school marching band.

The Related Post

In recent years, much attention has been paid to setting up Test Automation frameworks which are effective, easy to maintain, and allow the whole testing team to contribute to the testing effort. In doing so, we often leave out one of the most critical considerations of Test Automation: What do we do when the Test ...
Recently while teaching a workshop on Testing Dirty Systems, I uttered this “Randyism” off the top of my head, “Test automation is not automatic.” I realized immediately that I had just concisely stated the problem in making test automation a reality in many organizations. Most testers know that test automation is not automatic. (Wouldn’t it be great?) However, ...
We’ve scoured the internet to search for videos that provide a wealth of knowledge about Test Automation. We curated this short-list of videos that cover everything from the basics, to the more advanced, and why Test Automation should be part of part of any software development organization. Automation Testing Tutorial for Beginners This tutorial introduces ...
One of my current responsibilities is to find ways to automate, as much as practical, the ‘testing’ of the user experience (UX) for complex web-based applications. In my view, full test automation of UX is impractical and probably unwise; however, we can use automation to find potential UX problems, or undesirable effects, even in rich, ...
This article was developed from concepts in the book Global Software Test Automation: A Discussion of Software Testing for Executives, by Hung Q. Nguyen, Michael Hacket and Brent K. Whitlock Introduction The top 5 pitfalls encountered by managers employing software Test Automation are: Uncertainty and lack of control Poor scalability and maintainability Low Test Automation ...
People who know me and my work probably know my emphasis on good test design for successful test automation. I have written about this in “Key Success Factors for Keyword Driven Testing“. In the Action Based Testing (ABT) method that I have pioneered over the years it is an essential element for success. However, agreeing ...
A short-list of selection criteria and popular automation tools. There are a lot of test automation tools available in the market, from heavy-duty enterprise level tools to quick and dirty playback-and-record tools for browser testing. For anyone just starting their research we’ve put together a short list of requirements and tools to consider.
“Happy About Global Software Test Automation: A Discussion of Software Testing for Executives” Author: Hung Q. Nguyen, Michael Hackett, and Brent K. Whitlock Publisher: Happy About (August 1, 2006) Finally, a testing book for executives!, November 17, 2006 By Scott Barber “Chief Technologist, PerfTestPlus” Happy About Global Software Test Automation: A Discussion of Software Testing ...
Test Automation is significant and growing-yet I have read many forum comments and blog posts about Test Automation not delivering as expected. It’s true that test automation can improve reliability while minimizing variability in the results, speed up the process, increase test coverage, and ultimately provide greater confidence in the quality of the software being ...
Identifying which tests to begin with when starting automation is key to driving testing cycle times down and coverage up. So there you are. You’ve done a little research and made the business case to upper management regarding test automation and they bit on the proposal. Surprisingly, they supported you all the way and are extremely ...
LogiGear Magazine January Trends Issue 2017
Introduction As a consultant and trainer, I am often asked by my clients and students how to deal with automated acceptance tests. One common question is whether automated acceptance tests should use the graphical user interface (GUI) provided by the application.

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news