Successfully reported this slideshow.
Your SlideShare is downloading. ×

End-to-end dialogue systems, or a feature which wasn’t meant to happen | Rasa Summit

End-to-end dialogue systems, or a feature which wasn’t meant to happen | Rasa Summit

Download to read offline

You know the feeling when you ask for something and you’re pretty sure “no” will be the answer, but you still do it, because why not try? Well… the story of end-to-end is exactly this! Before starting on it, we read several papers about the technology not being ready for end-to-end dialogues in production. So, when we started working on it as a research project, “negative results are also interesting results” was our mantra. Suddenly, the results started to look more and more promising. Then, we developed the end-to-end training further – so that one can combine the classic Rasa format with intents and actions with the new end-to-end and gradually get rid of intents they don’t need.

In short, I will tell you a story of how end-to-end grew from a little internship project into an experimental feature of Rasa (and spanned far beyond the internship).

Presented by Evgeniia Razumovskaia, PhD on Computation, Cognition and Language at University of Cambridge at the 2021 Rasa Summit https://rasa.com/summit/

You know the feeling when you ask for something and you’re pretty sure “no” will be the answer, but you still do it, because why not try? Well… the story of end-to-end is exactly this! Before starting on it, we read several papers about the technology not being ready for end-to-end dialogues in production. So, when we started working on it as a research project, “negative results are also interesting results” was our mantra. Suddenly, the results started to look more and more promising. Then, we developed the end-to-end training further – so that one can combine the classic Rasa format with intents and actions with the new end-to-end and gradually get rid of intents they don’t need.

In short, I will tell you a story of how end-to-end grew from a little internship project into an experimental feature of Rasa (and spanned far beyond the internship).

Presented by Evgeniia Razumovskaia, PhD on Computation, Cognition and Language at University of Cambridge at the 2021 Rasa Summit https://rasa.com/summit/

More Related Content

More from Rasa Technologies

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

End-to-end dialogue systems, or a feature which wasn’t meant to happen | Rasa Summit

  1. 1. End-to-end training, or a feature which wasn’t meant to happen Evgeniia (Genie) Razumovskaia PhD on Computation, Cognition and Language, University of Cambridge
  2. 2. End-to-end training ● … as a new feature of Rasa ● … as an example of how research proceeds within Rasa
  3. 3. What’s on: ● What is end-to-end training? ● Why end-to-end? Why now? ● What was the research process? ● What happens after research? (my biggest surprise 😅)
  4. 4. What is end-to-end training? Traditional pipeline End-to-end training: predict the response text directly from the preceding text Image from here with little modifications
  5. 5. What is end-to-end training? - story: end to end happy path steps: - user: “hi” - bot: “hi!” - user: “I’m looking for a restaurant” - bot: “how about Chinese food?” - user: “sure” - bot: “here’s what I found ...” Input data: just like dialogues in books: Both user’s and bot’s responses are defined by words they “say” (not abstracted representations, e.g., intents ) Example from here
  6. 6. What is end-to-end training? One model from text in context to next bot response text
  7. 7. Why end-to-end? Why not? + 1 model for all instead of 3 + no need to annotate data with intents, slots, action labels and possible responses + data is more legible - we need a lot of data to make end-to-end work properly - we have less control over the outputs of the model - story: end to end happy path steps: - user: “hi” - bot: “hi!” - user: “I’m looking for a restaurant” - bot: “how about Chinese food?” - user: “sure” - bot: “here’s what I found ...”
  8. 8. Why now? ● On the one hand, fully unsupervised learning is seen as infeasible in production ● On the other hand, it is time that we start getting rid of intents in chatbots Well… CHALLENGE ACCEPTED! 😋 - story: why we need end-to-end steps: - user: greet - bot: utter_greet - user: search_restaurant - bot: utter_suggest_cuisine - user: “I had that yesterday” - bot: utter_suggest_cuisine - story: why we need end-to-end steps: - user: greet - bot: utter_greet - user: search_restaurant - bot: utter_suggest_cuisine - user: “I had that yesterday but I am always up for Thai” - bot: utter_search_restaurant Examples from here
  9. 9. What are main stages of research process at Rasa? Research raw idea🤓 If successful, integrate the idea into Rasa 👩‍🔧 Welcome the users to use it as an experimental feature 🎉
  10. 10. 1st stage: Research raw idea🤓 Convince ourselves that end-to-end works overall ● Run experiments on multidomain text-to-text dataset in English -- MultiWOZ (Budzianowski et al, 2018) ● As Rasa retrieves the responses (rather than generating it), pick 20 random options ● Results looked good, so, we decided to proceed Go through steps that our users would ● Took a bot built with a traditional pipeline and started “getting rid” of intents (Sara) ● Chatted with it as users would;
  11. 11. Model Hits@1 (20 candidates) [Mehri et al., 2019]1 67.34 Ours 71.8 [Mehri et al., 2019]2 72.18
  12. 12. 2nd stage: integrate the idea into Rasa 👩‍🔧 ● Going fully end-to-end is a radical change of approach to how you keep your training data, which models need to be trained etc. ⇒ How to combine these 2 approaches?
  13. 13. 2nd stage: integrate the idea into Rasa 👩‍🔧 Example from here - story: fully old format steps: - intent: greet - action: utter_greet - intent: search_restaurant - action: utter_suggest_cuisine - intent: affirm - action: utter_search_restaurant Usual “old” format - story: fully old format steps: - intent: greet - action: utter_greet - intent: search_restaurant - action: utter_suggest_cuisine - user: “I am always up for sushi” - action: utter_search_restaurant “Old” format with some user turns in text (mix) - story: end to end story steps: - user: “hi” - bot: “hi!” - user: “I’m looking for a restaurant” - bot: “how about Chinese food?” - user: “sure” - bot: “here’s what I found ...” Fully end-to-end format
  14. 14. 2nd stage: integrate the idea into Rasa 👩‍🔧 - story: fully old format steps: - intent: greet - action: utter_greet - intent: search_restaurant - action: utter_suggest_cuisine - user: “I am always up for sushi” Mixed dialogue ● we have NLU data mapping texts to intents; ● we have end-to-end stories, so, dialogue policy will be able to take text into account in prediction 1. NLU model predicts an intent Example from here How the prediction works 2. Dialogue policy can look directly at the text 3. Dialogue policy decides that the intent doesn’t contain useful information and predicts next action directly from text search_restaurant
  15. 15. 2nd stage: integrate the idea into Rasa 👩‍🔧 What does it mean? ● This makes intents optional ● We have a powerful dialogue policy which can be trained on intents and action names, texts of phrases of the user and bot and any mix of those After a couple of turns around the spiral of “this works but isn’t technically feasible” and “this is technically beautiful but doesn’t work”...
  16. 16. Welcome the users to use it as an experimental feature 🎉 Welcome! End-to-end is an experimental feature in Rasa 2.2
  17. 17. Thank you for your attention! @erazumovskaia evgeniar@yahoo.com

×