The guide for CUI Automated Testing strategies, including chatbot testing and voice app testing.
In the Software Testing industry, trends come and go that shape the future of testing. From Automation in Agile, to the DevOps era we are now in, trends are what evolve and improve our testing processes and ideologies. Currently, many researchers believe the next big thing in the testing industry will be the use of artificial intelligence (AI) in conversational user interfaces. In fact, a 2017 study by Forrester forecasted that 25% of jobs will be impacted by AI technologies (including intelligent chatbots) by 2019—a frightening premonition for some, but an open door for those workers who are currently intrigued by the notion of skilling up. But before we dive into that, we need to first understand what a conversational UI is.
What is a Conversational UI?
A conversational user interface (CUI) allows for users to interact with the computer and/or software in a human-like manner. It has evolved from the past versions of communication between humans and computers, where pre-defined scripts were used to respond to customer questions via an instant messenger-esque setup. Nowadays, these UIs mimic conversations one would have with an actual human being. There are 2 overarching types of conversational UIs: (1) a voice assistant that allows you to talk (i.e. Amazon’s Alexa, Google’s “OK Google,” Microsoft’s Cortana, or Apple’s Siri) and (2) chatbots that allow you to type.
Types of Conversational UIs
Voice Assistants
These types of conversational UIs complete actions by having users simply speak a command. Apple was one of the first companies to have their voice assistant, Siri, widely adopted; launched in October 2011, Siri provided iPhone users the ability to access information and complete actions on their device by simply saying, “Hey Siri.” Voice assistants have only been developed further since then; we now have home devices, such as Google Home and Amazon Echo, that allow users to not only access copious amounts of information but also control aspects of their home, including turning the lights on and off, locking and unlocking doors and windows, even purchasing items from the Amazon marketplace simply by using their voice!
Chatbots
Chatbots are mobile or web-based interfaces that allow the user to ask questions and retrieve information. They conduct a conversation via auditory or textual methods. Typically, chatbots are used in dialog systems—a computer system intended to converse with a human—for a variety of tasks, including customer service or acquiring information.
How do CUIs Work?
Many times, conversational interfaces use natural language processing (NLP), a division of artificial intelligence that helps computers to understand and interpret human language. Although natural language processing takes into account the structure of human language (words make a phrase, phrases have meanings, etc.), the ambiguity of human language sometimes baffles NLP. This has led to a shift towards natural language understanding (NLU). NLU attempts to aid in machine reading comprehension by allowing a line of questioning to continue throughout the conversation; it allows the machine to handle unstructured inputs that a human brain could successfully interpret (i.e. spelling errors, follow-up questions, etc.), thus streamlining the entire process of CUIs. While many chatbots use NLP and NLU systems, many others simply scan the inputted query for keywords, and then pull a reply from a database that has the most matching keywords.
For example, at LogiGear, we have a chatbot on our TestArchitect Support website. From the backend, we trained our bot by letting it learn our content from the TestArchitect Online Help Knowledge Base. Developers could see the chat log between the bot and the user and would keyword map any related article with the conversation for the bot to learn. By doing this, the next time a user asks a question with the same keywords, the bot can suggest the article on its own accord. You can refer to Figure 1 for a visual depiction of how this works.
Chatbot Testing Strategies
Chatbot testing falls under the denomination of UI/UX testing, which is on the highest level of the testing hierarchy; this is because when you are testing a chatbot, you are testing essentially the entire system and all of the dependent aspects of the system. Despite it being a high level test, the UI of a chatbot is relatively simple: it’s either spoken or written human language. Thus, a test case here would be a conversation that the chatbot should be able to follow. As far as a test suite for chatbots, this usually consists of a library of conversation transcripts. When creating the test suite, you’ll want to identify what Chatbots Magazine describes as the “Happy Paths,” where the user behaves as expected and continue with the other conversation flows. This way, whenever one of the conversations fails, you will easily be able to identify the error.
Running the test suite manually is quite easy—start Facebook Messenger, open the chatbot, and start chatting! However, this is only feasible for small projects as it can quickly become time consuming and redundant. That is where Test Automation comes in handy. With Automation, you can run your chatbot conversations without you actually driving the conversation; instead, it is essentially a chatbot interacting with another chatbot. All of the tedious work will be completed by the bots, and all you have to do is wait to receive the test reports.
But, this approach only checks to see if the chatbot is continuing the conversation, not if the chatbot actually fulfills the requirements. The approach mentioned above is a verification approach (as opposed to a validation approach); essentially, this method of Automation is checking if the chatbot is actually working as designed, and we assume that it will do the right thing.
Why Test Chatbots?
A quick look into some comments and reviews on early Amazon Alexa products will answer this question for you. From “it asks me if I’m ready but won’t proceed,” to “it doesn’t acknowledge my correct answers!”—it’s pretty evident why we need to test CUIs. One of the biggest challenges in CUIs is the fact that there are no barriers for users; this means that users, either with chatbots or voice activated systems, have absolute freedom to say whatever they wish—meaning CUIs have to cover all kinds of unexpected user inputs. Here’s an example of what I mean:
User: CUI, should I bring a jacket today?
CUI: Where are you going today?
User: Chocolate ice cream!
CUI: …
In this scenario, we’d want the bot to either (a) redirect the conversation or (b) ask the question again. It’s important to design your chatbot test cases with the expectation of unexpected user input. It’s also important to understand a 100% testing coverage rate for CUIs is impossible; because the nature of human language and all of the infinitely possible inputs, we as testers need to plan to cover the most common situations. For instance, having your chatbot know the conversion of cups to liters seems reasonable, but knowing the lifespan of the orchid mantis insect may be unnecessary. We at LogiGear are quite familiar with testing and automating voice apps, including Amazon’s Alexa. We have an entire article about the ins and outs of testing voice assistants, as well as a step-by-step guide and a video. Written by Do Nguyen, you can check it out here.
Importance of CUIs
For Businesses
Chatbots are becoming increasingly prevalent in businesses, as they automate tasks that do not require skill-based talents. There are a plethora of reasons for the implementation of chatbots, one of the most prevalent being efficiency. It’s no secret that software is faster and less prone to errors than humans when it comes to completing tasks. Moreover, these bots can handle astronomically more demands and pieces of information at once than humans.
Time and cost efficiency are definitely beneficial for businesses, but an even larger benefit is the customer service improvement. One study found 53% of customers that ask a question on Twitter expect a response within the hour; having an automated chatbot that works alongside and supports the support team by answering questions 24/7 not only pleases the customers—but also alleviates the pressure these teams may feel. More importantly, it cuts down waiting time for customers with non-specific issues and allows those who actually need the support of a technical support person to experience shorter wait times. Econsultancy reports that 45% of financial service firms are focused on making the customer experience as relevant and precise as possible—and Automation can help with that.
For Software Testers
It’s no secret that chatbots are going to kill jobs; as stated previously in 2017, Forrester forecasted that 25% of jobs will be impacted by AI technologies. But, similar to the first law of thermodynamics—that states that energy can neither be created nor destroyed but rather transformed into a different form—the unskilled labor jobs that are being overtaken are instead being replaced with new jobs and possibilities. A study by Modis found that the tech industry will grow by at least 12% by 2024, and a LinkedIn survey from 2017 confirmed this, finding that business and activities in the tech industry have a growing need for talent. We have an entire article dedicated to this topic, titled Hack Your Test Team’s Productivity, so be sure to check that out!
For the Industry
It is widely known that AI is becoming an increasingly hot topic for all industries, but none more important than the tech industry—this is the industry that builds AI! In fact, Artificial Solutions, one of the industry leaders in enterprise-strength conversational AI, won 4 Stevie 2019 International Business Awards—one of the world’s most coveted prizes for businesses. They took the silver Stevie in the very prestigious Most Innovative Tech Company of the Year, which recognizes a company’s achievement in product innovation. Their software, Teneo, has truly raised the bar for competition conversational AI platforms. Microsoft is one of these companies trying to keep themselves at the forefront of AI innovation; they recently released an updated version of their digital assistant, Cortana, which boasts a new chat-based UI. As opposed to single queries, the new version will show the entire extent of the interaction with Cortana as a means of making the interaction more conversational and more seamless.
One of the most thrilling aspects of what conversational UI means for the Software Testing industry is the unknown aspect of what is to come. It’s obvious that new jobs are going to be created for tech developers that are both ready and willing to work alongside these new technologies. The intelligent virtual assistant (IVA) market—a related field to conversational UIs—is expected to reach a value of $25.63 billion USD by 2025, with a compound annual growth rate of 40.4%. Despite these huge figures, the market is actually expected to grow aggressively over forecasts.
Summary
The growth of conversational UIs is both exciting yet worrisome: new technologies are going to drastically improve efficiency in the workplace, improve customer satisfaction, and reduce error—but at the expense of jobs. Yet, we as a global society have already faced a similar workplace situation with the Industrial Revolution; laborers at the time were terrified of what was to come. But, with open minds and growing education, people found ways to work alongside new technology. They say history repeats itself, so hopefully humanity reacts in the same way: uncovering new jobs and industries to work alongside technology. So long as we can do so, conversational UIs will continue to develop and improve business processes worldwide.
