Conversational User Interfaces: Breaking Down the Best Automated Testing Strategies

The guide for CUI Automated Testing strategies, including chatbot testing and voice app testing.

In the Software Testing industry, trends come and go that shape the future of testing. From Automation in Agile, to the DevOps era we are now in, trends are what evolve and improve our testing processes and ideologies. Currently, many researchers believe the next big thing in the testing industry will be the use of artificial intelligence (AI) in conversational user interfaces. In fact, a 2017 study by Forrester forecasted that 25% of jobs will be impacted by AI technologies (including intelligent chatbots) by 2019—a frightening premonition for some, but an open door for those workers who are currently intrigued by the notion of skilling up. But before we dive into that, we need to first understand what a conversational UI is.

What is a Conversational UI?

A conversational user interface (CUI) allows for users to interact with the computer and/or software in a human-like manner. It has evolved from the past versions of communication between humans and computers, where pre-defined scripts were used to respond to customer questions via an instant messenger-esque setup. Nowadays, these UIs mimic conversations one would have with an actual human being. There are 2 overarching types of conversational UIs: (1) a voice assistant that allows you to talk (i.e. Amazon’s Alexa, Google’s “OK Google,” Microsoft’s Cortana, or Apple’s Siri) and (2) chatbots that allow you to type.

Types of Conversational UIs

Voice Assistants

These types of conversational UIs complete actions by having users simply speak a command. Apple was one of the first companies to have their voice assistant, Siri, widely adopted; launched in October 2011, Siri provided iPhone users the ability to access information and complete actions on their device by simply saying, “Hey Siri.” Voice assistants have only been developed further since then; we now have home devices, such as Google Home and Amazon Echo, that allow users to not only access copious amounts of information but also control aspects of their home, including turning the lights on and off, locking and unlocking doors and windows, even purchasing items from the Amazon marketplace simply by using their voice!


Chatbots are mobile or web-based interfaces that allow the user to ask questions and retrieve information. They conduct a conversation via auditory or textual methods. Typically, chatbots are used in dialog systems—a computer system intended to converse with a human—for a variety of tasks, including customer service or acquiring information.

How do CUIs Work?

Many times, conversational interfaces use natural language processing (NLP), a division of artificial intelligence that helps computers to understand and interpret human language. Although natural language processing takes into account the structure of human language (words make a phrase, phrases have meanings, etc.), the ambiguity of human language sometimes baffles NLP. This has led to a shift towards natural language understanding (NLU). NLU attempts to aid in machine reading comprehension by allowing a line of questioning to continue throughout the conversation; it allows the machine to handle unstructured inputs that a human brain could successfully interpret (i.e. spelling errors, follow-up questions, etc.), thus streamlining the entire process of CUIs. While many chatbots use NLP and NLU systems, many others simply scan the inputted query for keywords, and then pull a reply from a database that has the most matching keywords.

Figure 1 –  Example of the TA chatbot process

For example, at LogiGear, we have a chatbot on our TestArchitect Support website. From the backend, we trained our bot by letting it learn our content from the TestArchitect Online Help Knowledge Base. Developers could see the chat log between the bot and the user and would keyword map any related article with the conversation for the bot to learn. By doing this, the next time a user asks a question with the same keywords, the bot can suggest the article on its own accord. You can refer to Figure 1 for a visual depiction of how this works.

Chatbot Testing Strategies

Chatbot testing falls under the denomination of UI/UX testing, which is on the highest level of the testing hierarchy; this is because when you are testing a chatbot, you are testing essentially the entire system and all of the dependent aspects of the system. Despite it being a high level test, the UI of a chatbot is relatively simple: it’s either spoken or written human language. Thus, a test case here would be a conversation that the chatbot should be able to follow. As far as a test suite for chatbots, this usually consists of a library of conversation transcripts. When creating the test suite, you’ll want to identify what Chatbots Magazine describes as the “Happy Paths,”  where the user behaves as expected and continue with the other conversation flows. This way, whenever one of the conversations fails, you will easily be able to identify the error.

Running the test suite manually is quite easy—start Facebook Messenger, open the chatbot, and start chatting! However, this is only feasible for small projects as it can quickly become time consuming and redundant. That is where Test Automation comes in handy. With Automation, you can run your chatbot conversations without you actually driving the conversation; instead, it is essentially a chatbot interacting with another chatbot. All of the tedious work will be completed by the bots, and all you have to do is wait to receive the test reports.

But, this approach only checks to see if the chatbot is continuing the conversation, not if the chatbot actually fulfills the requirements. The approach mentioned above is a verification approach (as opposed to a validation approach); essentially, this method of Automation is checking if the chatbot is actually working as designed, and we assume that it will do the right thing.

Why Test Chatbots?

A quick look into some comments and reviews on early Amazon Alexa products will answer this question for you. From “it asks me if I’m ready but won’t proceed,” to “it doesn’t acknowledge my correct answers!”—it’s pretty evident why we need to test CUIs. One of the biggest challenges in CUIs is the fact that there are no barriers for users; this means that users, either with chatbots or voice activated systems, have absolute freedom to say whatever they wish—meaning CUIs have to cover all kinds of unexpected user inputs. Here’s an example of what I mean:

User: CUI, should I bring a jacket today?

CUI: Where are you going today?

User: Chocolate ice cream!

CUI: …

In this scenario, we’d want the bot to either (a) redirect the conversation or (b) ask the question again. It’s important to design your chatbot test cases with the expectation of unexpected user input. It’s also important to understand a 100% testing coverage rate for CUIs is impossible; because the nature of human language and all of the infinitely possible inputs, we as testers need to plan to cover the most common situations. For instance, having your chatbot know the conversion of cups to liters seems reasonable, but knowing the lifespan of the orchid mantis insect may be unnecessary. We at LogiGear are quite familiar with testing and automating voice apps, including Amazon’s Alexa. We have an entire article about the ins and outs of testing voice assistants, as well as a step-by-step guide and a video. Written by Do Nguyen, you can check it out here.

Importance of CUIs

For Businesses

Chatbots are becoming increasingly prevalent in businesses, as they automate tasks that do not require skill-based talents. There are a plethora of reasons for the implementation of chatbots, one of the most prevalent being efficiency. It’s no secret that software is faster and less prone to errors than humans when it comes to completing tasks. Moreover, these bots can handle astronomically more demands and pieces of information at once than humans.

Time and cost efficiency are definitely beneficial for businesses, but an even larger benefit is the customer service improvement. One study found 53% of customers that ask a question on Twitter expect a response within the hour; having an automated chatbot that works alongside and supports the support team by answering questions 24/7 not only pleases the customers—but also alleviates the pressure these teams may feel. More importantly, it cuts down waiting time for customers with non-specific issues and allows those who actually need the support of a technical support person to experience shorter wait times. Econsultancy reports that 45% of financial service firms are focused on making the customer experience as relevant and precise as possible—and Automation can help with that.

For Software Testers

It’s no secret that chatbots are going to kill jobs; as stated previously in 2017, Forrester forecasted that 25% of jobs will be impacted by AI technologies. But, similar to the first law of thermodynamics—that states that energy can neither be created nor destroyed but rather transformed into a different form—the unskilled labor jobs that are being overtaken are instead being replaced with new jobs and possibilities. A study by Modis found that the tech industry will grow by at least 12% by 2024, and a LinkedIn survey from 2017 confirmed this, finding that business and activities in the tech industry have a growing need for talent. We have an entire article dedicated to this topic, titled Hack Your Test Team’s Productivity, so be sure to check that out!

For the Industry

It is widely known that AI is becoming an increasingly hot topic for all industries, but none more important than the tech industry—this is the industry that builds AI! In fact, Artificial Solutions, one of the industry leaders in enterprise-strength conversational AI, won 4 Stevie 2019 International Business Awards—one of the world’s most coveted prizes for businesses. They took the silver Stevie in the very prestigious Most Innovative Tech Company of the Year, which recognizes a company’s achievement in product innovation. Their software, Teneo, has truly raised the bar for competition conversational AI platforms. Microsoft is one of these companies trying to keep themselves at the forefront of AI innovation; they recently released an updated version of their digital assistant, Cortana, which boasts a new chat-based UI. As opposed to single queries, the new version will show the entire extent of the interaction with Cortana as a means of making the interaction more conversational and more seamless.

One of the most thrilling aspects of what conversational UI means for the Software Testing industry is the unknown aspect of what is to come. It’s obvious that new jobs are going to be created for tech developers that are both ready and willing to work alongside these new technologies. The intelligent virtual assistant (IVA) market—a related field to conversational UIs—is expected to reach a value of $25.63 billion USD by 2025, with a compound annual growth rate of 40.4%. Despite these huge figures, the market is actually expected to grow aggressively over forecasts.


The growth of conversational UIs is both exciting yet worrisome: new technologies are going to drastically improve efficiency in the workplace, improve customer satisfaction, and reduce error—but at the expense of jobs. Yet, we as a global society have already faced a similar workplace situation with the Industrial Revolution; laborers at the time were terrified of what was to come. But, with open minds and growing education, people found ways to work alongside new technology. They say history repeats itself, so hopefully humanity reacts in the same way: uncovering new jobs and industries to work alongside technology. So long as we can do so, conversational UIs will continue to develop and improve business processes worldwide.

Noah Peters
Noah Peters is from the Bay Area. He holds a Bachelor's Degree in Marketing from the University of San Francisco. Noah started at LogiGear as a Marketing Intern and transitioned to a full-time Marketing Associate role post-graduation. Noah is passionate about content creation and SEO, and works closely with all content produced by LogiGear, including the LogiGear Magazine, the LogiGear Blog, and various LogiGear eBooks. In his free time, you can find Noah researching the automotive industry or teaching high school marching band.

The Related Post

Investing in Test Automation training will increase your team’s productivity. The availability of reliable jobs in a competitive US market seems to be constantly embattled with competition and replacements of artificial intelligence (AI). In 2016, Foxconn replaced 60,000 employees with robots. However, the growth of Test Automation as an occupation has highlighted an intriguing option ...
The growing complexity of the Human-Machine Interface (HMI) in cars offers traditional testers an opportunity to capitalize on their strengths. The human-machine interface (HMI) is nothing new. Any user interface including a graphical user interface (GUI) falls under the category of human-machine interface. HMI is more commonly being used to mean a view into the ...
When configured with a Python harness, TestArchitect can be used to automate testing on software for custom hardware Unlike other proprietary and open source tools, that are able to automate only desktop, or mobile, TestArchitect (TA Test) has the ability to test the software that runs on hardware in the following ways: 1. TA can ...
Introduction Many executives have some very basic questions about Software Testing. These questions address the elements of quality (customer satisfaction) and money (spending the least amount of money to prevent future loss). The basic questions that executive have about Software Testing include: Why care about and spend money on testing? Why should testing be treated ...
How lagging automotive design principles adversely affect final products. Cars are integrating more and more software with every model year. The ginormous screen introduced by Tesla in their flagship Model S a few years ago was seemingly unrivaled at the time. Nowadays, screens of this size are not only commonplace in vehicles such as the ...
As I wrote in various articles, organization is one of the 3 key requisites for successful automated testing, the other two being test design and automation architecture.
People who know me and my work probably know my emphasis on good test design for successful test automation. I have written about this in “Key Success Factors for Keyword Driven Testing“. In the Action Based Testing (ABT) method that I have pioneered over the years it is an essential element for success. However, agreeing ...
For this interview, we talked to Greg Wester, Senior Member Technical Staff, Craig Jennings, Senior Director, Quality Engineering and Ritu Ganguly, QE Director at Salesforce. is a cloud-based enterprise software company specializing in software as a service (SaaS). Best known for its Customer Relationship Management (CRM) product, it was ranked number 27 in Fortune’s 100 ...
Introduction As a consultant and trainer, I am often asked by my clients and students how to deal with automated acceptance tests. One common question is whether automated acceptance tests should use the graphical user interface (GUI) provided by the application.
The 12 Do’s and Don’ts of Test Automation When I started my career as a Software Tester a decade ago, Test Automation was viewed with some skepticism.
Every once in a while a book is put together that should be read by every person with a relationship to software development. This book is one of them. Everyone dreams of automating their software testing, but few make it a reality. This down-to-earth book contains stories of 28 teams that went for it, including ...
How to do UI test automation with the fewest headaches I’m currently interviewing lots of teams that have implemented acceptance testing for my new book. A majority of those interviewed so far have at some point shot themselves in the foot with UI test automation. After speaking to several people who are about to do ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news