This document discusses Conversation-Driven Development (CDD) as a process for building conversational AI assistants. It describes CDD as involving six key actions: sharing prototypes with test users, reviewing user conversations, annotating conversations as training data, testing the assistant, tracking the success of conversations, and fixing issues found. CDD captures lessons learned from building conversational AI to help teams iteratively improve assistants based on user feedback. The document also promotes Rasa X as a tool that can help teams implement CDD through features like reviewing conversations, annotating data, and continuous integration testing.

1. Conversation-Driven
Development
Alan Nichol | co-founder, CTO @ Rasa
July 2020

2. Agenda
CONVERSATION DRIVEN DEVELOPMENT
● About Rasa
● Conversation Driven Development
● CDD in Practice
● Q&A
Housekeeping
● Please keep yourself muted and your camera turned off
● Use the Zoom chat to ask questions. We’ll answer
questions at the end, but you can ask your questions at
any point during the broadcast
● We’ll post the recorded webinar on YouTube in a few
days. We’ll email you the link along with a short survey.

3. Pop Quiz! 🍿
rasa.com/cdd-quiz

4. About Rasa

5. We’re building the standard infrastructure for conversational AI
HOW WE ACHIEVE OUR MISSION
Champion Open
Source Tools
Nurture a Community
of Makers
Ship Applied Research to
Solve Real Problems

6. Developers all over the world build with Rasa:
2m+ downloads, 9,500+ forum members, 400+ contributors
RASA COMMUNITY
Healthcare Insurance Banking
Telecommunications Manufacturing
Technology

7. 5 levels of
conversational AI

8. The 5 levels of conversational AI got a 2020 upgrade

9. All of you are already gathering the data you need
to climb from level 3 to 4 and 5.
You just need to listen.

10. Conversation-Driven Development:
the process of listening to your users and using
those insights to improve your AI assistant

11. Conversation-Driven
Development (CDD)

12. CDD captures the lessons we’ve learned as a community
WHY CDD
If you’ve built conversational AI before, you know that:
● It is very hard.
● Building a prototype is not the hard part.
● The hard parts all show up when you want to go from a
prototype to something you’d want to ship.
Conversation-Driven Development should:
● Help all of us build better conversational AI.
● Save newcomers from having to learn this the hard way.

13. Conversation-Driven Development is made up of six actions
ABOUT CDD
share fix
track
review annotate test

14. Users will always surprise you.
So get some test users to try your prototype as early
as possible.
Shipping without having a bunch of test
users has never worked. Your project
won’t be the exception.
Share

15. Review
At every stage of a project, it is worth reading what users are
saying.
Avoid getting caught up in metrics right away. Conversations
are valuable data.

16. Annotate
Using a script to generate
synthetic training data
Turning real messages into
training examples

17. Test
Professional teams don’t ship applications without
tests.
Use whole conversations as end-to-end tests
Run them on a continuous integration (CI) server.

18. Prototype
Track
Use proxy measures to track which conversations are
successful and which ones failed.
‘Negative’ signals are useful too, e.g. users not getting back in
touch with support.

19. Prototype
Fix
Study conversations that went smoothly and ones that failed.
Successful conversations can become new tests 🎉
Fix issues by annotating more data and/or fixing your code 🔧

20. CDD in practice

21. It’s not a linear process: you’ll find yourself jumping between these actions
PROCESS
share fix
track
review annotate test

22. Some actions require software skills, others a deep understanding of the user
TEAM
Image credit: Freepik

23. Rasa X is a free tool to
help teams do CDD

24. ≈≈
Rasa X layers on top of Rasa Open Source
PRODUCT
Rasa X: Tool for
Conversation-Driven Development
Rasa Open Source: Framework for
building AI assistants

25. Rasa X local mode

26. Rasa X server mode

27. Share your bot with testers using just a link

28. Review conversations coming in from every channel

29. Annotate the messages coming in

30. #232a2e
Push new training data to git and trigger your CI pipeline

31. Track failures and successes
Use the API to automatically tag conversations, or add
tags manually as you read
Turn successful conversations into new
end-to-end tests

32. Q&A

33. Recap

34. CDD captures the lessons we’ve learned as a community
WHY CDD
Conversational AI is hard - CDD helps us make our assistants better
over time.
As experienced conversational AI teams, we owe it to newcomers to
help them avoid mistakes.
That’s the only way this field can continue to progress.

35. Join the Conversation

36. Share your knowledge and help other teams move faster
rasa.com/cdd-quiz

37. Alan Nichol
Co-founder & CTO
alan@rasa.com
Email me!