In this talk, Rachael Tatman, Senior Developer Advocate at Rasa, talks about how to continuously bringing humans into the development process to improve conversational AI systems. Presented on 09/29/2020 **These slides are from a talk given at Rsqrd AI. Learn more at rsqrdai.org**

1. Making Conversational AI Work for
Everybody: An Intro to Conversation
Driven Development
Dr. Rachael Tatman

2. Quick About Me

3. How I got here
About Me
● 2012- 2017: PhD in Linguistics
○ Modelling demographic disparities in Automatic
Speech Recognition
○ FAT* ML
● 2017- 2020: Developer Advocate @ Kaggle (Google)
○ "The home of data science"
○ Machine learning education
○ Live Streaming: Livecoding & Paper Reading
● 2020 - now: Developer Advocate @ Rasa
○ Open source conversational AI framework & free
software for conversation driven development
○ Developer education
○ Video, blogs, tutorials, live streaming, talks :)

4. Sooo.... why do I choose
to work on chatbots/
conversational AI?

5. Some false assumptions we tend to make
- Everyone has access to a computer
- Everyone has access to the internet
- Everyone knows how to Google stuff
- Everyone likes visual interfaces
- Everyone has a good/affordable enough
internet connection to load graphics
- Everyone knows how to use a website
- Everyone knows how to use YOUR website
- Everyone can make calls during business
hours
https://answers.microsoft.com/en-us/ie/forum/ie11-iewindows8_1/how-much-data-being-us
ed-when-visiting-a-website/5028b804-3adf-4093-b10a-a058d020b19a

6. Why conversational AI?
- Most people already know how to have a conversation
- NLP methods are able to handle a lot more variation in
input now
- A focused Conversational AI can help people do tasks
much faster
- Chalmers, a conversational assistant for people experiencing
homelessness, helped people find information 60 times
faster than searching (https://www.amplelabs.co/chalmers/)

7. Why conversational AI?
- Most people already know how to have a conversation
- NLP methods are able to handle a lot more variation in
input now
- A focused Conversational AI can help people do tasks
much faster
- Chalmers, an converational assistant for people experiencing
homelessness, helped people find information 60 times faster
than searching
Your assistant has to hold up their end of the bargain:
have a good, helpful conversation that meets your users’
expectations and genuinely helps them.
Good design is important, but you probably can’t predict
everything your users will need/want/do.

8. Conversation-Driven
Development (CDD)

9. CDD captures the lessons we’ve learned as a community
WHY CDD
If you’ve built conversational AI before, you know that:
● It is very hard.
● Building a prototype is not the hard part.
● The hard parts all show up when you want to go from a
prototype to something you’d want to ship.
Conversation-Driven Development should:
● Help all of us build better conversational AI.
● Save newcomers from having to learn this the hard way.

10. Conversation-Driven Development is made up of six actions
ABOUT CDD
share fixtrackreview annotate test

11. Users will always surprise you.
So get some test users to try your prototype as early
as possible.
Shipping without having a bunch of test
users has never worked. Your project
won’t be the exception.
Share

12. Review
At every stage of a project, it is worth reading what users are
saying.
Avoid getting caught up in metrics right away. Conversations
are valuable data.

13. Annotate
Using a script to generate
synthetic training data
Turning real messages into
training examples

14. Test
Professional teams don’t ship applications without
tests.
Use whole conversations as end-to-end tests
Run them on a continuous integration (CI) server.

15. PrototypeTrack
Use proxy measures to track which conversations are
successful and which ones failed.
‘Negative’ signals are useful too, e.g. users not getting back in
touch with support.

16. PrototypeFix
Study conversations that went smoothly and ones that failed.
Successful conversations can become new tests 🎉
Fix issues by annotating more data and/or fixing your code 🔧

17. CDD in practice

18. It’s not a linear process: you’ll find yourself jumping between these actions
PROCESS
share fixtrackreview annotate test

19. Some actions require software skills, others a deep understanding of the user
TEAM
Image credit: Freepik

20. How does this help?

21. Why conversational AI?
- Most people already know how to have a conversation
- NLP methods are able to handle a lot more variation in
input now
- A focused Conversational AI can help people do tasks
much faster
- Chalmers, an converational assistant for people experiencing
homelessness, helped people find information 60 times faster
than searching
Your assistant has to hold up their end of the bargain:
have a good, helpful conversation that meets your users’
expectations and genuinely helps them.
Good design is important, but you probably can’t predict
everything your users will need/want/do.

22. Why does CDD help?
- CDD lets you meet users where they
are now and help them right away
- You don’t have to predict or guess
what your users want or need
- You can make quick, incremental
improvements
- You can meet shifting needs or
expectations

23. Rachael Tatman
@rctatman
r.tatman@rasa.com
Rasa Technologies GmbH, Schönhauser Allee 175, 10119 Berlin, Germany
Questions?