vodQA Gurgaon 2019

1. Shama Ugale
Testing Conversational AI

2. What Do You Expect from this
session?

3. Agenda
• What, Why and How of a Conversational AI?
• Challenges
• Test Approaches for a Conversational AI
• Test types
• Use Case – A Coffee Shop Bot
• Automation with Botium
• Measuring the Quality

4. What is a Conversational AI?
Conversational AI systems are voice or text based messaging tool between a human and a robot, which
uses Artificial Intelligence and natural language processing to interact with the humans.

5. Few Examples

6. Why do we need
to test chatbots?

7. Tay – A Microsoft Bot on Twitter

8. Alexa – Home Automation Solution
Alexa owner returned
home to find his Alexa
started a 'party' at 2am,
leading to police
breaking down his door

9. Poncho – A Weather Bot

10. CNN – A News Bot

11. Challenges
• Self learning systems
• Non-linear input
• Non-deterministic user interactions
• No barriers for users

12. How do I test it?

13. A Dialogflow Chatbot Architecture

14. A Use
Case –
Coffee
Shop
Bot

15. Test Approach
MANUAL
TESTING
AUTOMATION
TESTING
CROWD
TESTING

16. Manual Testing

17. Types of testing
• Personality and on-boarding
• Intelligence
• Error management
• Understanding
• Speed and Accuracy
• Navigation

18. Automation
Testing – Botium

19. Botium in Nutshell

20. Test Setup
• Botium Box
• Dialogflow Agent

21. Demo

22. Measuring the Quality
• Goal completion rate
• Self-service rate
• Error rates
• AI and ML rates
• User retention rates

23. Questions

24. References
• Photos: © Google, Giphy
• https://chatbotsmagazine.com/
• https://chatbotslife.com/
• https://www.botium.at
• https://chatbottest.com
• http://www.botium.at/start.html
• https://medium.com/@floriantreml/botium-in-a-nutshell-part-1-overview-
f8d0ceaf8fb4