Conversational User Interfaces: Breaking Down the Best Automated Testing Strategies

The guide for CUI Automated Testing strategies, including chatbot testing and voice app testing.

In the Software Testing industry, trends come and go that shape the future of testing. From Automation in Agile, to the DevOps era we are now in, trends are what evolve and improve our testing processes and ideologies. Currently, many researchers believe the next big thing in the testing industry will be the use of artificial intelligence (AI) in conversational user interfaces. In fact, a 2017 study by Forrester forecasted that 25% of jobs will be impacted by AI technologies (including intelligent chatbots) by 2019—a frightening premonition for some, but an open door for those workers who are currently intrigued by the notion of skilling up. But before we dive into that, we need to first understand what a conversational UI is.

What is a Conversational UI?

A conversational user interface (CUI) allows for users to interact with the computer and/or software in a human-like manner. It has evolved from the past versions of communication between humans and computers, where pre-defined scripts were used to respond to customer questions via an instant messenger-esque setup. Nowadays, these UIs mimic conversations one would have with an actual human being. There are 2 overarching types of conversational UIs: (1) a voice assistant that allows you to talk (i.e. Amazon’s Alexa, Google’s “OK Google,” Microsoft’s Cortana, or Apple’s Siri) and (2) chatbots that allow you to type.

Types of Conversational UIs

Voice Assistants

These types of conversational UIs complete actions by having users simply speak a command. Apple was one of the first companies to have their voice assistant, Siri, widely adopted; launched in October 2011, Siri provided iPhone users the ability to access information and complete actions on their device by simply saying, “Hey Siri.” Voice assistants have only been developed further since then; we now have home devices, such as Google Home and Amazon Echo, that allow users to not only access copious amounts of information but also control aspects of their home, including turning the lights on and off, locking and unlocking doors and windows, even purchasing items from the Amazon marketplace simply by using their voice!


Chatbots are mobile or web-based interfaces that allow the user to ask questions and retrieve information. They conduct a conversation via auditory or textual methods. Typically, chatbots are used in dialog systems—a computer system intended to converse with a human—for a variety of tasks, including customer service or acquiring information.

How do CUIs Work?

Many times, conversational interfaces use natural language processing (NLP), a division of artificial intelligence that helps computers to understand and interpret human language. Although natural language processing takes into account the structure of human language (words make a phrase, phrases have meanings, etc.), the ambiguity of human language sometimes baffles NLP. This has led to a shift towards natural language understanding (NLU). NLU attempts to aid in machine reading comprehension by allowing a line of questioning to continue throughout the conversation; it allows the machine to handle unstructured inputs that a human brain could successfully interpret (i.e. spelling errors, follow-up questions, etc.), thus streamlining the entire process of CUIs. While many chatbots use NLP and NLU systems, many others simply scan the inputted query for keywords, and then pull a reply from a database that has the most matching keywords.

Figure 1 –  Example of the TA chatbot process

For example, at LogiGear, we have a chatbot on our TestArchitect Support website. From the backend, we trained our bot by letting it learn our content from the TestArchitect Online Help Knowledge Base. Developers could see the chat log between the bot and the user and would keyword map any related article with the conversation for the bot to learn. By doing this, the next time a user asks a question with the same keywords, the bot can suggest the article on its own accord. You can refer to Figure 1 for a visual depiction of how this works.

Chatbot Testing Strategies

Chatbot testing falls under the denomination of UI/UX testing, which is on the highest level of the testing hierarchy; this is because when you are testing a chatbot, you are testing essentially the entire system and all of the dependent aspects of the system. Despite it being a high level test, the UI of a chatbot is relatively simple: it’s either spoken or written human language. Thus, a test case here would be a conversation that the chatbot should be able to follow. As far as a test suite for chatbots, this usually consists of a library of conversation transcripts. When creating the test suite, you’ll want to identify what Chatbots Magazine describes as the “Happy Paths,”  where the user behaves as expected and continue with the other conversation flows. This way, whenever one of the conversations fails, you will easily be able to identify the error.

Running the test suite manually is quite easy—start Facebook Messenger, open the chatbot, and start chatting! However, this is only feasible for small projects as it can quickly become time consuming and redundant. That is where Test Automation comes in handy. With Automation, you can run your chatbot conversations without you actually driving the conversation; instead, it is essentially a chatbot interacting with another chatbot. All of the tedious work will be completed by the bots, and all you have to do is wait to receive the test reports.

But, this approach only checks to see if the chatbot is continuing the conversation, not if the chatbot actually fulfills the requirements. The approach mentioned above is a verification approach (as opposed to a validation approach); essentially, this method of Automation is checking if the chatbot is actually working as designed, and we assume that it will do the right thing.

Why Test Chatbots?

A quick look into some comments and reviews on early Amazon Alexa products will answer this question for you. From “it asks me if I’m ready but won’t proceed,” to “it doesn’t acknowledge my correct answers!”—it’s pretty evident why we need to test CUIs. One of the biggest challenges in CUIs is the fact that there are no barriers for users; this means that users, either with chatbots or voice activated systems, have absolute freedom to say whatever they wish—meaning CUIs have to cover all kinds of unexpected user inputs. Here’s an example of what I mean:

User: CUI, should I bring a jacket today?

CUI: Where are you going today?

User: Chocolate ice cream!

CUI: …

In this scenario, we’d want the bot to either (a) redirect the conversation or (b) ask the question again. It’s important to design your chatbot test cases with the expectation of unexpected user input. It’s also important to understand a 100% testing coverage rate for CUIs is impossible; because the nature of human language and all of the infinitely possible inputs, we as testers need to plan to cover the most common situations. For instance, having your chatbot know the conversion of cups to liters seems reasonable, but knowing the lifespan of the orchid mantis insect may be unnecessary. We at LogiGear are quite familiar with testing and automating voice apps, including Amazon’s Alexa. We have an entire article about the ins and outs of testing voice assistants, as well as a step-by-step guide and a video. Written by Do Nguyen, you can check it out here.

Importance of CUIs

For Businesses

Chatbots are becoming increasingly prevalent in businesses, as they automate tasks that do not require skill-based talents. There are a plethora of reasons for the implementation of chatbots, one of the most prevalent being efficiency. It’s no secret that software is faster and less prone to errors than humans when it comes to completing tasks. Moreover, these bots can handle astronomically more demands and pieces of information at once than humans.

Time and cost efficiency are definitely beneficial for businesses, but an even larger benefit is the customer service improvement. One study found 53% of customers that ask a question on Twitter expect a response within the hour; having an automated chatbot that works alongside and supports the support team by answering questions 24/7 not only pleases the customers—but also alleviates the pressure these teams may feel. More importantly, it cuts down waiting time for customers with non-specific issues and allows those who actually need the support of a technical support person to experience shorter wait times. Econsultancy reports that 45% of financial service firms are focused on making the customer experience as relevant and precise as possible—and Automation can help with that.

For Software Testers

It’s no secret that chatbots are going to kill jobs; as stated previously in 2017, Forrester forecasted that 25% of jobs will be impacted by AI technologies. But, similar to the first law of thermodynamics—that states that energy can neither be created nor destroyed but rather transformed into a different form—the unskilled labor jobs that are being overtaken are instead being replaced with new jobs and possibilities. A study by Modis found that the tech industry will grow by at least 12% by 2024, and a LinkedIn survey from 2017 confirmed this, finding that business and activities in the tech industry have a growing need for talent. We have an entire article dedicated to this topic, titled Hack Your Test Team’s Productivity, so be sure to check that out!

For the Industry

It is widely known that AI is becoming an increasingly hot topic for all industries, but none more important than the tech industry—this is the industry that builds AI! In fact, Artificial Solutions, one of the industry leaders in enterprise-strength conversational AI, won 4 Stevie 2019 International Business Awards—one of the world’s most coveted prizes for businesses. They took the silver Stevie in the very prestigious Most Innovative Tech Company of the Year, which recognizes a company’s achievement in product innovation. Their software, Teneo, has truly raised the bar for competition conversational AI platforms. Microsoft is one of these companies trying to keep themselves at the forefront of AI innovation; they recently released an updated version of their digital assistant, Cortana, which boasts a new chat-based UI. As opposed to single queries, the new version will show the entire extent of the interaction with Cortana as a means of making the interaction more conversational and more seamless.

One of the most thrilling aspects of what conversational UI means for the Software Testing industry is the unknown aspect of what is to come. It’s obvious that new jobs are going to be created for tech developers that are both ready and willing to work alongside these new technologies. The intelligent virtual assistant (IVA) market—a related field to conversational UIs—is expected to reach a value of $25.63 billion USD by 2025, with a compound annual growth rate of 40.4%. Despite these huge figures, the market is actually expected to grow aggressively over forecasts.


The growth of conversational UIs is both exciting yet worrisome: new technologies are going to drastically improve efficiency in the workplace, improve customer satisfaction, and reduce error—but at the expense of jobs. Yet, we as a global society have already faced a similar workplace situation with the Industrial Revolution; laborers at the time were terrified of what was to come. But, with open minds and growing education, people found ways to work alongside new technology. They say history repeats itself, so hopefully humanity reacts in the same way: uncovering new jobs and industries to work alongside technology. So long as we can do so, conversational UIs will continue to develop and improve business processes worldwide.

Noah Peters
Noah Peters is from the Bay Area. He holds a Bachelor's Degree in Marketing from the University of San Francisco. Noah started at LogiGear as a Marketing Intern and transitioned to a full-time Marketing Associate role post-graduation. Noah is passionate about content creation and SEO, and works closely with all content produced by LogiGear, including the LogiGear Magazine, the LogiGear Blog, and various LogiGear eBooks. In his free time, you can find Noah researching the automotive industry or teaching high school marching band.

The Related Post

There are few topics in quality assurance testing that cause as much confusion as smoke testing versus sanity testing. The two names would seem to describe very different practices— and they do! But people still get them confused, since the distinction is somewhat subtle.
The challenges with any automation effort is to know your capability. I’ve seen too many automation efforts begin and end with a tool decision. Generally these tools are very complex pieces of software that do many more things then we would ever use in our normal everyday testing. It even adds more misery to the ...
June Issue 2019: Testing the Software Car
For this interview, we talked to Greg Wester, Senior Member Technical Staff, Craig Jennings, Senior Director, Quality Engineering and Ritu Ganguly, QE Director at Salesforce. is a cloud-based enterprise software company specializing in software as a service (SaaS). Best known for its Customer Relationship Management (CRM) product, it was ranked number 27 in Fortune’s 100 ...
LogiGear Magazine January Trends Issue 2017
LogiGear Magazine – April 2014 – Test Tool and Automation
This book isn’t for everyone, but everyone can get some value out of it. What I mean by that rather confusing statement is that folks working in Agile environments will likely want to throw the book across the room while folks in more bureaucratic environments like CMMI or other waterfall environments will likely get a ...
When automated tests are well-organized and written with the necessary detail, they can be very efficient and maintainable. But designing automated tests that deal with data can be challenging if you have a lot of data combinations. For example, let’s say we want to simulate a series of 20 customers, along with the number of ...
Introduction Many executives have some very basic questions about Software Testing. These questions address the elements of quality (customer satisfaction) and money (spending the least amount of money to prevent future loss). The basic questions that executive have about Software Testing include: Why care about and spend money on testing? Why should testing be treated ...
We’ve scoured the internet to search for videos that provide a wealth of knowledge about Test Automation. We curated this short-list of videos that cover everything from the basics, to the more advanced, and why Test Automation should be part of part of any software development organization. Automation Testing Tutorial for Beginners This tutorial introduces ...
We’re celebrating the 1st birthday of our Agile eBook! It has been one year since we launched our eBook on Agile Automation. To celebrate, we’ve updated the foreword and included a brand new automation checklist! As we take the moment to mark this occasion, we wanted to take some time to reflect on the State ...
Understanding the benefits and challenges of Automating ERP is critical. According to SAP, ERP (Enterprise Resource Planning) “is the core processes that are needed to run a company: finance, human resources, manufacturing, supply chain, services, procurement, and others. At its most basic level, ERP integrates these processes into a single system. But new ERP systems ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news