Successfully reported this slideshow.
Your SlideShare is downloading. ×

How to Effectively Test Your Chatbot | Rasa Summit

How to Effectively Test Your Chatbot | Rasa Summit

Download to read offline

QA has always been under-rated and thus it is important to consider this equally important as the Dev. If we look at the Chatbot QA, it had been considered as a highly challenging work specially when you do not know where your bot may break while you sequentially will be only running your flow (stories). Most of the companies / tools only check the flow which are coded in a fixed format which often breaks while testing. There may be cases where bot are migrated to new version and it breaks. The presentation will discuss the possibilities to test the bots by helping folks to create their coverage matrix for your stories, efficiently looking at the logs and mine information and most importantly what to test and which components to test.

Presented by Director QA, DevOps & AIML at APTY.IO, Soumya Mukherjee at the 2021 Rasa Summit https://rasa.com/summit/

QA has always been under-rated and thus it is important to consider this equally important as the Dev. If we look at the Chatbot QA, it had been considered as a highly challenging work specially when you do not know where your bot may break while you sequentially will be only running your flow (stories). Most of the companies / tools only check the flow which are coded in a fixed format which often breaks while testing. There may be cases where bot are migrated to new version and it breaks. The presentation will discuss the possibilities to test the bots by helping folks to create their coverage matrix for your stories, efficiently looking at the logs and mine information and most importantly what to test and which components to test.

Presented by Director QA, DevOps & AIML at APTY.IO, Soumya Mukherjee at the 2021 Rasa Summit https://rasa.com/summit/

More Related Content

More from Rasa Technologies

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

How to Effectively Test Your Chatbot | Rasa Summit

  1. 1. How to ”Effectively” ”Test” your Chatbot Soumya Mukherjee Director QA, DevOps & AIML Apty.IO
  2. 2. How are we doing our QA today • Testing is Blackbox for testers • Mostly manual testing done in organization • Conversational flow testing • Small Talk • Fallback checks • Integrations • Automation done on UI and API layer • Testing is mostly done on same training data • Models are trained by engineers and are not being monitored by QA • There are analytics tools available to monitor but it needs technical expertise for the QA • Result : More than 90% times bot breaks (no one understands when it will break), most of them fallback and get stuck - once bot is stuck it is stuck Q ? A
  3. 3. What are the issues in QA ? • Bots are evolving and continuous story creation is a problem • No tool manage story coverage • Your training data may not correspond to new stories or vice versa (it’s a mismatch) – most org keep training on the same data • Most automation tools offers record and playback (My stories are already written how to port is the question)
  4. 4. What are the issues in QA ? • No (unified) centralized dashboard present where QA can check (everything is quite scattered) • Intent Matching • Entity Testing – Slot identification • Entity Testing – Entity Validation • Confidence score • Confusion Matrix along with Precision/Recall/F1-Score • No easy way to reset the failed bot ! • Bot versioning is a mess and A/B testing becomes difficult • Multilingual bot QA is a challenge (have to make 2 separate bots) • High confidence score is also a problem as your bot will only predict same thing (if the data is same for multiple intents then it will predict the one with highest confidence score – may be incorrect) How to make sure your bot never breaks ?
  5. 5. How to make your test effective ? • Create scenarios for happy path, contextual questions, digressions, domain specific questions, stateless conversations • Map proper entities for common scenarios (example bus fee, tuition fee) – flow should change with entities in the stories • Automated tests should consume all stories and run them each time as part of regression testing • Story coverage visualization • For Manual Testing use Bot emulation product (like RasaX, Botfront) to test
  6. 6. How to make your test effective ? • Central dashboarding including : • Confusion matrix, Precision, Recall and F1-Score • Cumulative accuracy profile • Cross validation results • Perform Exhaustive testing (bot resiliency), Integration checks across platforms, Webhooks • Perform fault tolerance testing by performing performance testing (bot response, session management) & security testing (api interaction, typing speed check, punctuations, typo errors)
  7. 7. Other KPIs to track • Activity Volume • Bounce rate • Retention rate • Open sessions count • Session times (conversation length) • Goal completion rate • User feedback (sentiments) • Fallback rate (Confusion rate, reset rate & Human takeover rate)
  8. 8. Thanks @QASoumya Linkedin.com/in/mukherjeesoumya

×