Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LDN BotFramework Meetup #3: Robots Testing Robots

2,167 views

Published on

Kristian Brimble and Sam Kavanagh, dev and test leads on the JustEat chatbot, take us through the automated testing strategy they used

Published in: Software
  • Be the first to comment

  • Be the first to like this

LDN BotFramework Meetup #3: Robots Testing Robots

  1. 1. Robots Testing Robots @LDNBotFramework ❖ Kristian Brimble / Sam Kavanagh ❖ Bot Team: Dev Lead / Lead QA ❖ @kristianbrimble / @justeat_tech
  2. 2. Just Eat Help Chatbot
  3. 3. What does it do? 3 ● Provides customers with support when they have problems with their orders ● Can compensate customers with account credit when something was missing from their order ● Hands over to a human when it gets stuck
  4. 4. How does it work? Microsoft Directline Native App Bot Framework LUIS Junior Flows Principal Flows
  5. 5. Testing our bot
  6. 6. When, where and how? Unit Tests Run on developer machines on demand (NCrunch FTW) and on every check-in & build Integration Tests Can be run on developer machines. Mainly run as part of our automated deployment process System Tests Can be run on developer machines but only targeting staging environment. Automatically run when a build is deployed to staging UI Tests Run on app developer machines and as part of their CI pipeline
  7. 7. Unit Testing ● Originally started unit tests only on services called as part of a conversation ● Dialogs are hard to test due to difficulty to mock the framework and large amounts of setup required (very brittle) ● Separated “conversation flow” from dialogs ● Conversation flow classes are very self-contained and do not reference any Bot Framework features
  8. 8. Integration Testing ● Calls to Just Eat dependencies are proxied by a service that returns pre-recorded responses ● We forked the Bot Framework emulator so that it could be used by automated tests ○ No more dependency on the Bot Framework being contactable! ○ Reduced time to run our test suite by a factor of 10 ● We wrote a small BDD-style framework to make testing a conversational interface easier (more on that later)
  9. 9. System Testing ● Still uses our hacked emulator and BDD-style framework ● No dependencies mocked - “real data” created on our staging environment ● Smaller number of tests that exercise as much of a conversation topic as possible in one test
  10. 10. UI Testing ● Automated iOS and Android UI tests ● Written in each platform’s native language (Swift and Java) ● Run when the native bot client’s change ● All calls to Direct Line are mocked so that only UI functionality is tested
  11. 11. Example of a system test
  12. 12. Any questions?
  13. 13. Thanks to: @LDNBotFramework ❖ Just Eat ❖ Jamie Dalton @daltskin [Handing off to a human] ❖ Kristin Brimble @kristianbrimble ❖ Sam Kavanagh @justeat_tech [Robots Testing Robots] ❖ Robin Osborne @rposbo [Speaker Recognition API]

×