The document outlines upcoming features and improvements for Rasa Open Source, including a new YAML training data format, incremental training, automated testing, and enhanced machine learning model inspection. It emphasizes the importance of breaking free from traditional intents to create more adaptive conversational AI and the need for continuous model evaluation. Additionally, controllable natural language generation is highlighted as a future goal to provide context-aware responses.

1. Rasa Open Source - What’s next?
Rasa Summit 2021
Tom Bocklisch, Director of Engineering @ Rasa
February 2021

2. Rasa Open Source
New Training Data Format
Rule Policy
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases

3. YAML Training Data Format
Rasa Open Source 2.0
https://rasa.com/docs/rasa/training-data-format

4. YAML Training Data Format
Rasa Open Source 2.0
Why did we do this?
● Well-known standard
● Good editor support
● Unified extensible format
● Composable
money_transfer:
- I want to transfer money
- I want to make a transfer
Financial bot
loan_queries:
- I need a loan
- I want to take out a loan
https://rasa.com/docs/rasa/training-data-format

5. Rasa Open Source
New Training Data Format
Rule Policy
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases
New Documentation

6. New Documentation including
a brand new Playground
Rasa Open Source 2.0
Why did we do this?
● Better structure
● Completeness
● No-install quick start
https://rasa.com/docs/rasa/playground

7. Rasa Open Source
New Training Data Format
Rule Policy
Incremental Training
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases
New Documentation

8. Incremental Training
Rasa Open Source 2.2
https://blog.rasa.com/rasa-new-incremental-training/
Why did we do this?
● Iterate quickly
● On-the-spot retraining
● Composability

9. Rasa Open Source
New Training Data Format
Rule Policy
Incremental Training
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases
New Documentation
Automated Testing

10. Automated Testing of Assistants
Rasa Open Source 2.2
Why did we do this?
● Ship with confidence
● Easy to set up
● Follow SE best practices
https://github.com/marketplace?query=rasahq

11. Rasa Open Source
New Training Data Format
Rule Policy
Incremental Training
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases
Machine Learning
Model Inspection
New Documentation
Automated Testing

12. Machine Learning Model Inspection
Rasa Open Source 2.3
Why did we do this?
● Customization
● Research
● Debuggability
https://github.com/rasahq/rasalit

13. Future of Rasa Open Source
So, what is next then?

14. Rasa Open Source
New Training Data Format
Rule Policy
Incremental Training
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases
Machine Learning
Model Inspection
Identifying Unsuccessful
Conversations at Scale
New Documentation
Automated Testing

15. Where do users drop off? Why do users drop off? What can I do to fix that?
Identifying Unsuccessful Conversations at Scale
FEATURE PREVIEW
More powerful, proactive filters
(e.g. notifications to review new conversations where a
fallback action was triggered, free text search, etc.)
Incorporate emerging research (IntentTED)
(e.g. detect where an unexpected intent was expressed at a point
in the conversation)

16. Rasa Open Source
New Training Data Format
Rule Policy
Incremental Training
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases
Machine Learning
Model Inspection
Identifying Unsuccessful
Conversations at Scale
Continuous Model
Evaluation
New Documentation
Automated Testing

17. Continuous Model Evaluation methodology for CDD
FEATURE PREVIEW
What do we want to achieve?
● The ability to compare versions of
a bot despite the fact that they
participated in different
conversations.
● Providing a quantifiable measure
of progress.
● Estimating and evaluating the
impact of a new
feature/model/model component.
Second Bot
Version
First Bot
Version

18. Rasa Open Source
New Training Data Format
Rule Policy
Incremental Training
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases
Machine Learning
Model Inspection
Identifying Unsuccessful
Conversations at Scale
Continuous Model
Evaluation
Breaking free
from Intents
New Documentation
Automated Testing

19. Adaptive Assistants will never come if everyone
sticks with designing and implementing
hypothetical conversations, training on synthetic
data, etc.

20. To get to level 3 and beyond we have to reinvent how conversational AI works.
LEVEL 3 AND BEYOND
What you expect: What users also do:

21. Rasa Open Source
New Training Data Format
Rule Policy
Incremental Training
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases
Machine Learning
Model Inspection
Identifying Unsuccessful
Conversations at Scale
Continuous Model
Evaluation
Breaking free
from Intents
Experimental
e2e Training
New Documentation
Automated Testing

22. Get end-to-end ready for prime time. Breaking free of intents is just the start.
FEATURE PREVIEW
All of these changes will make it easier to modify models or integrate research ideas.
We are aiming to make end-to-end
part of every assistant.
Still some heavy lifting ahead:
● Refactor Structure of
Dialogue Events
● Refactor Tracker Store
Persistence
● Refactor Agent / Processor /
Tracker Store

23. Rasa Open Source
New Training Data Format
Rule Policy
Incremental Training
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases
Machine Learning
Model Inspection
Identifying Unsuccessful
Conversations at Scale
Continuous Model
Evaluation
Breaking free
from Intents
Experimental
e2e Training
Controllable NLG
New Documentation
Automated Testing

24. Controllable Natural Language Generation (NLG)
FEATURE PREVIEW
What do we want to achieve?
● Adapt responses to their context →
currently possible with custom logic in
an action → does not scale
● NLG model that learns from examples
but guarantees transparent control
for conversational designers and
developers (longer term)
What does this look like?
1. Mirror users word choices
● If user uses “sneakers” assistant uses “sneakers”
● If user uses “runners” assistant uses “runners”
● If user uses “kicks” assistant uses “kicks”
2. Adjusting response based on previous responses
● First time: “I’m sorry. I don’t understand. Could you
try rephrasing?”
● Second time: “I’m very sorry. I don’t quite get what
you mean. Would you like to speak to customer
service?”
● Third time: “My apologies. Let’s hand you over to a
customer service agent, ...”

25. Rasa Open Source
New Training Data Format
Rule Policy
Incremental Training
Last Major Release
2.0
Current Version
2.3
Upcoming
Releases
Machine Learning
Model Inspection
Identifying Unsuccessful
Conversations at Scale
Continuous Model
Evaluation
Breaking free
from Intents
Experimental
e2e Training
Controllable NLG
New Documentation
Automated Testing

26. Tom Bocklisch
Director of Engineering
tom@rasa.com
@tombocklisch
Excited? Help us make
these ideas become reality!
rasa.com/jobs
👩‍💻 👨‍🚀 🚀