The document discusses testing conversational AI systems. It begins with defining conversational AI as voice or text-based systems that interact with humans using natural language processing. It then discusses challenges in testing these systems, such as their self-learning abilities and non-linear user interactions. The document outlines various test approaches, including testing the system's personality, intelligence, error handling, understanding, speed, and navigation abilities. It also discusses different types of testing and metrics for measuring quality. Finally, it provides examples of conversational systems and references additional testing tools and frameworks.

1. Testing Conversational AI
Shama Ugale
Hyderabad 16th March 2019
Seismic Shifts

2. 2
Shama Ugale
Senior Consultant - Quality Analyst at ThoughtWorks, Inc
9+ years of experience (as QA & Solution Consultant)
About Me

3. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 3
Agenda
• What is a Conversational AI?
• Challenges in Testing a Conversational AI
• Test Approaches for a Conversational AI
• Test types
• Measuring the Quality
• Testing Tools and frameworks

4. ©ThoughtWorks 2017 Commercial in Confidence
What Do You Expect from this session?
4

5. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 5
Era of AI

6. 6
What is a Conversational AI?
Conversational AI systems are voice or text based messaging tool between a human and a robot, which
uses Artificial Intelligence and natural language processing to interact with the humans.

7. 7
Why have Conversational systems gained
the popularity?

8. “People are now spending more time in
messaging apps than in social media and
that is a huge turning point. Messaging
apps are the platforms of the future and
bots will be how their users access all sorts
of services.” —Peter Rojas, Entrepreneur in
Residence at Betaworks
8

10. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 10
Few Examples

11. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 11
How are bots helping businesses?
● Personalization
● 24*7 availability, Bots never sleep
● More Customer Engagement
● Cost Effective
● Less training and faster on-boarding
● Alternative Sales Channel

12. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 12
Types of Conversational Systems
Rule Based
These bots answer questions based on some rules on which it is trained on. The rules defined
can be very simple to very complex.
AI Based
These bots come with their own brains and ability to continuously learn on their own based on
the conversations they have in the past. They are implemented using Deep Learning algorithms and
Natural Language processing technologies.

13. ©ThoughtWorks 2017 Commercial in Confidence
13
Some success stories

14. ©ThoughtWorks 2017 Commercial in Confidence
14

15. ©ThoughtWorks 2017 Commercial in Confidence
Why do we need to test
Chatbots?
15

16. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 16
Tay – A Microsoft Bot on Twitter

17. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 17
Alexa – Home Automation Solution
Alexa owner returned home to find his Alexa started a 'party' at 2am, leading to police breaking down his
door

18. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 18
Poncho – The weather bot

19. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 19
CNN – News Bot

20. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 20
Challenges In testing Chatbots
• Self learning systems
• Non-linear input
• Non-deterministic user interactions
• No barriers for users

21. ©ThoughtWorks 2017 Commercial in Confidence
Test Approach
21

22. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 22
Personality and on-boarding
• Personality brings life to your chatbot user experience
• Attitude, style and tone, does it change over the time?
• Engaging, friendly and relevant to the context
• Meet and Greet
• Set Expectations
• Initiate small talks

23. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 23
Intelligence
• Context Remembrance and switching
• Multistep conversations
• Utterances

24. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 24
Error Management
• Conversational Errors
• Voice Recognition Errors
• Fail gracefully

25. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 25
Understanding
• Accepted inputs
• Small talk
• Bad and Love words
• Language
• Acronyms & Jargons
• Typing mistakes

26. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 26
Speed and Accuracy
• Response time
• Valid answer

27. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 27
Navigation
• Switching the context
• Getting back to the previous conversations

28. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 28
Types of testing
• Performance testing
• Security testing
• Multilingual testing
• Voice Based Testing
• Domain Testing

29. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 29
Measuring the Quality
• Goal completion rate
• Self-service rate
• Error rates
• AI and ML rates
• User retention rates

30. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 30
TESTING TOOLS FOR CHATBOTS

31. ©ThoughtWorks, Inc. Do not copy. Do not distribute. 31
References
• Photos: © Google, Giphy
• https://chatbotsmagazine.com/
• https://chatbotslife.com/
• https://www.botium.at
• https://chatbottest.com
• https://dimon.co

32. THANK YOU
Shama Ugale
32