1. How to
”Effectively” ”Test”
your Chatbot
Soumya Mukherjee
Director QA, DevOps & AIML
Apty.IO

2. How are we doing our QA today
• Testing is Blackbox for testers
• Mostly manual testing done in organization
• Conversational flow testing
• Small Talk
• Fallback checks
• Integrations
• Automation done on UI and API layer
• Testing is mostly done on same training data
• Models are trained by engineers and are not being
monitored by QA
• There are analytics tools available to monitor but it
needs technical expertise for the QA
• Result : More than 90% times bot breaks (no one
understands when it will break), most of them fallback
and get stuck - once bot is stuck it is stuck
Q ?
A

3. What are the issues in QA ?
• Bots are evolving and continuous story creation is a problem
• No tool manage story coverage
• Your training data may not correspond to new stories or vice versa (it’s a
mismatch) – most org keep training on the same data
• Most automation tools offers record and playback (My stories are
already written how to port is the question)

4. What are the issues in QA ?
• No (unified) centralized dashboard present where QA can check (everything is quite scattered)
• Intent Matching
• Entity Testing – Slot identification
• Entity Testing – Entity Validation
• Confidence score
• Confusion Matrix along with Precision/Recall/F1-Score
• No easy way to reset the failed bot !
• Bot versioning is a mess and A/B testing becomes difficult
• Multilingual bot QA is a challenge (have to make 2 separate bots)
• High confidence score is also a problem as your bot will only predict same thing (if the data is same
for multiple intents then it will predict the one with highest confidence score – may be incorrect)
How to make sure your bot never breaks ?

5. How to make your test effective ?
• Create scenarios for happy path, contextual questions, digressions, domain
specific questions, stateless conversations
• Map proper entities for common scenarios (example bus fee, tuition fee) –
flow should change with entities in the stories
• Automated tests should consume all stories and run them each time as part
of regression testing
• Story coverage visualization
• For Manual Testing use Bot emulation product (like RasaX, Botfront) to test

6. How to make your test effective ?
• Central dashboarding including :
• Confusion matrix, Precision, Recall and F1-Score
• Cumulative accuracy profile
• Cross validation results
• Perform Exhaustive testing (bot resiliency), Integration checks across
platforms, Webhooks
• Perform fault tolerance testing by performing performance testing (bot
response, session management) & security testing (api interaction,
typing speed check, punctuations, typo errors)

7. Other KPIs to track
• Activity Volume
• Bounce rate
• Retention rate
• Open sessions count
• Session times (conversation length)
• Goal completion rate
• User feedback (sentiments)
• Fallback rate (Confusion rate, reset rate & Human takeover rate)

8. Thanks
@QASoumya
Linkedin.com/in/mukherjeesoumya