Conversational User Interfaces: Breaking Down the Best Automated Testing Strategies

The guide for CUI Automated Testing strategies, including chatbot testing and voice app testing.

In the Software Testing industry, trends come and go that shape the future of testing. From Automation in Agile, to the DevOps era we are now in, trends are what evolve and improve our testing processes and ideologies. Currently, many researchers believe the next big thing in the testing industry will be the use of artificial intelligence (AI) in conversational user interfaces. In fact, a 2017 study by Forrester forecasted that 25% of jobs will be impacted by AI technologies (including intelligent chatbots) by 2019—a frightening premonition for some, but an open door for those workers who are currently intrigued by the notion of skilling up. But before we dive into that, we need to first understand what a conversational UI is.

What is a Conversational UI?

A conversational user interface (CUI) allows for users to interact with the computer and/or software in a human-like manner. It has evolved from the past versions of communication between humans and computers, where pre-defined scripts were used to respond to customer questions via an instant messenger-esque setup. Nowadays, these UIs mimic conversations one would have with an actual human being. There are 2 overarching types of conversational UIs: (1) a voice assistant that allows you to talk (i.e. Amazon’s Alexa, Google’s “OK Google,” Microsoft’s Cortana, or Apple’s Siri) and (2) chatbots that allow you to type.

Types of Conversational UIs

Voice Assistants

These types of conversational UIs complete actions by having users simply speak a command. Apple was one of the first companies to have their voice assistant, Siri, widely adopted; launched in October 2011, Siri provided iPhone users the ability to access information and complete actions on their device by simply saying, “Hey Siri.” Voice assistants have only been developed further since then; we now have home devices, such as Google Home and Amazon Echo, that allow users to not only access copious amounts of information but also control aspects of their home, including turning the lights on and off, locking and unlocking doors and windows, even purchasing items from the Amazon marketplace simply by using their voice!


Chatbots are mobile or web-based interfaces that allow the user to ask questions and retrieve information. They conduct a conversation via auditory or textual methods. Typically, chatbots are used in dialog systems—a computer system intended to converse with a human—for a variety of tasks, including customer service or acquiring information.

How do CUIs Work?

Many times, conversational interfaces use natural language processing (NLP), a division of artificial intelligence that helps computers to understand and interpret human language. Although natural language processing takes into account the structure of human language (words make a phrase, phrases have meanings, etc.), the ambiguity of human language sometimes baffles NLP. This has led to a shift towards natural language understanding (NLU). NLU attempts to aid in machine reading comprehension by allowing a line of questioning to continue throughout the conversation; it allows the machine to handle unstructured inputs that a human brain could successfully interpret (i.e. spelling errors, follow-up questions, etc.), thus streamlining the entire process of CUIs. While many chatbots use NLP and NLU systems, many others simply scan the inputted query for keywords, and then pull a reply from a database that has the most matching keywords.

Figure 1 –  Example of the TA chatbot process

For example, at LogiGear, we have a chatbot on our TestArchitect Support website. From the backend, we trained our bot by letting it learn our content from the TestArchitect Online Help Knowledge Base. Developers could see the chat log between the bot and the user and would keyword map any related article with the conversation for the bot to learn. By doing this, the next time a user asks a question with the same keywords, the bot can suggest the article on its own accord. You can refer to Figure 1 for a visual depiction of how this works.

Chatbot Testing Strategies

Chatbot testing falls under the denomination of UI/UX testing, which is on the highest level of the testing hierarchy; this is because when you are testing a chatbot, you are testing essentially the entire system and all of the dependent aspects of the system. Despite it being a high level test, the UI of a chatbot is relatively simple: it’s either spoken or written human language. Thus, a test case here would be a conversation that the chatbot should be able to follow. As far as a test suite for chatbots, this usually consists of a library of conversation transcripts. When creating the test suite, you’ll want to identify what Chatbots Magazine describes as the “Happy Paths,”  where the user behaves as expected and continue with the other conversation flows. This way, whenever one of the conversations fails, you will easily be able to identify the error.

Running the test suite manually is quite easy—start Facebook Messenger, open the chatbot, and start chatting! However, this is only feasible for small projects as it can quickly become time consuming and redundant. That is where Test Automation comes in handy. With Automation, you can run your chatbot conversations without you actually driving the conversation; instead, it is essentially a chatbot interacting with another chatbot. All of the tedious work will be completed by the bots, and all you have to do is wait to receive the test reports.

But, this approach only checks to see if the chatbot is continuing the conversation, not if the chatbot actually fulfills the requirements. The approach mentioned above is a verification approach (as opposed to a validation approach); essentially, this method of Automation is checking if the chatbot is actually working as designed, and we assume that it will do the right thing.

Why Test Chatbots?

A quick look into some comments and reviews on early Amazon Alexa products will answer this question for you. From “it asks me if I’m ready but won’t proceed,” to “it doesn’t acknowledge my correct answers!”—it’s pretty evident why we need to test CUIs. One of the biggest challenges in CUIs is the fact that there are no barriers for users; this means that users, either with chatbots or voice activated systems, have absolute freedom to say whatever they wish—meaning CUIs have to cover all kinds of unexpected user inputs. Here’s an example of what I mean:

User: CUI, should I bring a jacket today?

CUI: Where are you going today?

User: Chocolate ice cream!

CUI: …

In this scenario, we’d want the bot to either (a) redirect the conversation or (b) ask the question again. It’s important to design your chatbot test cases with the expectation of unexpected user input. It’s also important to understand a 100% testing coverage rate for CUIs is impossible; because the nature of human language and all of the infinitely possible inputs, we as testers need to plan to cover the most common situations. For instance, having your chatbot know the conversion of cups to liters seems reasonable, but knowing the lifespan of the orchid mantis insect may be unnecessary. We at LogiGear are quite familiar with testing and automating voice apps, including Amazon’s Alexa. We have an entire article about the ins and outs of testing voice assistants, as well as a step-by-step guide and a video. Written by Do Nguyen, you can check it out here.

Importance of CUIs

For Businesses

Chatbots are becoming increasingly prevalent in businesses, as they automate tasks that do not require skill-based talents. There are a plethora of reasons for the implementation of chatbots, one of the most prevalent being efficiency. It’s no secret that software is faster and less prone to errors than humans when it comes to completing tasks. Moreover, these bots can handle astronomically more demands and pieces of information at once than humans.

Time and cost efficiency are definitely beneficial for businesses, but an even larger benefit is the customer service improvement. One study found 53% of customers that ask a question on Twitter expect a response within the hour; having an automated chatbot that works alongside and supports the support team by answering questions 24/7 not only pleases the customers—but also alleviates the pressure these teams may feel. More importantly, it cuts down waiting time for customers with non-specific issues and allows those who actually need the support of a technical support person to experience shorter wait times. Econsultancy reports that 45% of financial service firms are focused on making the customer experience as relevant and precise as possible—and Automation can help with that.

For Software Testers

It’s no secret that chatbots are going to kill jobs; as stated previously in 2017, Forrester forecasted that 25% of jobs will be impacted by AI technologies. But, similar to the first law of thermodynamics—that states that energy can neither be created nor destroyed but rather transformed into a different form—the unskilled labor jobs that are being overtaken are instead being replaced with new jobs and possibilities. A study by Modis found that the tech industry will grow by at least 12% by 2024, and a LinkedIn survey from 2017 confirmed this, finding that business and activities in the tech industry have a growing need for talent. We have an entire article dedicated to this topic, titled Hack Your Test Team’s Productivity, so be sure to check that out!

For the Industry

It is widely known that AI is becoming an increasingly hot topic for all industries, but none more important than the tech industry—this is the industry that builds AI! In fact, Artificial Solutions, one of the industry leaders in enterprise-strength conversational AI, won 4 Stevie 2019 International Business Awards—one of the world’s most coveted prizes for businesses. They took the silver Stevie in the very prestigious Most Innovative Tech Company of the Year, which recognizes a company’s achievement in product innovation. Their software, Teneo, has truly raised the bar for competition conversational AI platforms. Microsoft is one of these companies trying to keep themselves at the forefront of AI innovation; they recently released an updated version of their digital assistant, Cortana, which boasts a new chat-based UI. As opposed to single queries, the new version will show the entire extent of the interaction with Cortana as a means of making the interaction more conversational and more seamless.

One of the most thrilling aspects of what conversational UI means for the Software Testing industry is the unknown aspect of what is to come. It’s obvious that new jobs are going to be created for tech developers that are both ready and willing to work alongside these new technologies. The intelligent virtual assistant (IVA) market—a related field to conversational UIs—is expected to reach a value of $25.63 billion USD by 2025, with a compound annual growth rate of 40.4%. Despite these huge figures, the market is actually expected to grow aggressively over forecasts.


The growth of conversational UIs is both exciting yet worrisome: new technologies are going to drastically improve efficiency in the workplace, improve customer satisfaction, and reduce error—but at the expense of jobs. Yet, we as a global society have already faced a similar workplace situation with the Industrial Revolution; laborers at the time were terrified of what was to come. But, with open minds and growing education, people found ways to work alongside new technology. They say history repeats itself, so hopefully humanity reacts in the same way: uncovering new jobs and industries to work alongside technology. So long as we can do so, conversational UIs will continue to develop and improve business processes worldwide.

Noah Peters
Noah Peters is from the Bay Area. He holds a Bachelor's Degree in Marketing from the University of San Francisco. Noah started at LogiGear as a Marketing Intern and transitioned to a full-time Marketing Associate role post-graduation. Noah is passionate about content creation and SEO, and works closely with all content produced by LogiGear, including the LogiGear Magazine, the LogiGear Blog, and various LogiGear eBooks. In his free time, you can find Noah researching the automotive industry or teaching high school marching band.

The Related Post

The path to continuous delivery leads through automation Software testing and verification needs a careful and diligent process of impersonating an end user, trying various usages and input scenarios, comparing and asserting expected behaviours. Directly, the words “careful and diligent” invoke the idea of letting a computer program do the job. Automating certain programmable aspects ...
< Michael Hackett sat down with EA’s Stephen Copp to discuss the world of integrated test platforms.
This book isn’t for everyone, but everyone can get some value out of it. What I mean by that rather confusing statement is that folks working in Agile environments will likely want to throw the book across the room while folks in more bureaucratic environments like CMMI or other waterfall environments will likely get a ...
Mobile usage today is not just a trend but it is an essential shift in how people communicate with each other, interact with the world, and do business. According to a ComScore, in 2014 the number of mobile users surpassed the number of computer users and is showing strong growth over time, towards some point in ...
*You can check the answer key here
Cross-Browser Testing is an integral part of the Software Testing world today. When we need to test the functionality of a website or web application, we need to do so on multiple browsers for a multitude of reasons.
Automated Testing is a huge part of DevOps, but without human-performed quality assurance testing, you’re increasing the risk of  lower-quality software making it into production.  Automated Testing is an essential DevOps practice to increase organizations’ release cadence and code quality. But there are definitely limits to only using Automated Testing. Without human quality assurance (QA) ...
September Issue 2018: The Secrets to Better Test Automation  
5 roadblocks in vehicular autonomy that complicate Software Testing Experts in the field have previously referred to air travel as somewhat of a gold standard for autonomous vehicle safety, but after Boeing’s two tragedies, that analogy can no longer be used when talking about self-driving cars. This was after Boeing’s 737 MAX Jets have found ...
Test Strategy A test strategy describes how the test effort will reach the quality goals set out by the development team. Sometimes called the test approach, test strategy includes, among other things, the testing objective, methods and techniques of testing and the testing environment.
Utility: A program that performs a specific task related to the management of computer functions, resources, or files, as password protection, memory management, virus protection, and file compression. Tool: A program or application that software development teams use to create, debug, maintain, or otherwise support other programs and applications. The term usually refers to programs that can be combined together ...
As I wrote in various articles, organization is one of the 3 key requisites for successful automated testing, the other two being test design and automation architecture.

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news