This document discusses testing strategies for chatbots like Bumblebee. It recommends using a mixture of manual and automated testing techniques. For manual testing, it suggests crowd testing without predefined rules to allow exploratory testing, as well as A/B testing. Automated testing should include regression, functional, conversation flow, smoke, and load tests. The test framework mentioned is Jest with a fluent interface using promises. Verification focuses on building functionality correctly while validation focuses on meeting requirements.

1. Chatbot testing
By Vijaykumar b

2. 2
What is Bumblebee?
No.. not this bee

3. 3
What is Bumblebee?
It’s pretty but not this one

4. 4
What is Bumblebee?
It’s a bot
It tells you about leaves
It raises tech-ops tickets for you

5. 5
I would say bumblebee is the personal assistant for every thought worker
Is that what bumblebee
can do?
Bumblebee is multi-skilled
Bumblebee understands thoughtworks and every thoughtworker

6. 6
Why Bumblebee is
personal-assistant?

7. 7

8. 8

9. 9

10. 10

11. 11
Bumblebee Architecture:

12. 12
What makes testing chatbot different?
Learning cloud services
Non-deterministic user interactions
No barriers for users

13. 13
Test Layer:

14. 14
How did we go about testing bumblebee?
There isn’t a single source of truth, but the mixture of testing techniques is the key to good software quality.
Make sure to understand the difference between “verification” and “validation”:
verification addresses the concern "are you building it right”
Focus on verification in automated testing (if chatbot is working as designed)
validation addresses the concern "are you building the right thing”
Focus on validation in manual testing (if chatbot fulfills requirements)

15. 15
Test Framework:
Jest
Fluent interface using JS promises
userSays(“hi”).botSays(“welcome to chatbot testing”)

16. 16
Manual Testing
Crowd Testing
Select the crowd wisely depending on the project state.
Dialect is the most important selection criteria and don't define any rules for the crowd to test.
Exploratory Testing
A/B Testing
same principle applies as crowd testing - don't define any rules to crowd
Let the crowd do their work
even when bot is live, you have to invest effort
Try difference utterances and measure changes in user behaviour

17. 17
Automation Testing
Regression Testing
Keep the chatbot's behaviour consistent as you make change
Dialogflow updates
Reverse engineering chatbot’s
Improve your chatbot functionality

18. 18
Functional Tests
End to end testing
Functioning of entire platform

19. 19
Conversation Flow tests
Maintain contexts
Maintain contexts
Change/break flow based on what user says

20. 20
Conversation flow tests

21. 21
Extend jest matchers to fit bot layout

22. 22
Smoke Test
Quick Feedback
Happy path in each sub-domain/feature

23. 23
Load Test
Lambdas
Dialogflow
chat.google.com

24. THANK YOU