Successfully reported this slideshow.

Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time

2

Share

1 of 44
1 of 44

Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time

2

Share

Download to read offline

Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time.
Ting-Hao K. Huang, Joseph Chee Chang, Jeffrey P. Bigham.
In Proceedings of Conference on Human Factors in Computing Systems 2018 (CHI 2018), 2018, Montréal, Canada.

Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time.
Ting-Hao K. Huang, Joseph Chee Chang, Jeffrey P. Bigham.
In Proceedings of Conference on Human Factors in Computing Systems 2018 (CHI 2018), 2018, Montréal, Canada.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time

  1. 1. 1 Live Note/QA: http://tinyurl.com/Evorus Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time Ting-Hao (Kenneth) Huang* Joseph Chee Chang Jeffrey P. Bigham [ Question / Feedback: http://tinyurl.com/Evorus ] *Kenneth will join in Fall 2018
  2. 2. 2 Live Note/QA: http://tinyurl.com/Evorus
  3. 3. 3 Live Note/QA: http://tinyurl.com/Evorus
  4. 4. 4 Live Note/QA: http://tinyurl.com/Evorus
  5. 5. 5 Live Note/QA: http://tinyurl.com/Evorus
  6. 6. 6 Live Note/QA: http://tinyurl.com/Evorus
  7. 7. 7 Live Note/QA: http://tinyurl.com/Evorus What just happened? • Open Conversation • Multi-turn interaction • Multiple domains • Personalized • Coherent dialog • Mix of task-oriented and social conversation
  8. 8. 8 Live Note/QA: http://tinyurl.com/Evorus Today’s Conversational Assistants… “What’s new with Alexa?”“Talking to Siri”
  9. 9. 9 Live Note/QA: http://tinyurl.com/Evorus Existing Approaches to Open Conversation • Combining multiple task-oriented dialog systems • DialPort (Zhao, et al., 2016) • End-to-end framework for dialogue systems • Serban, et al. 2016; Li, et al. 2017 • Adapting a model to many other domains • Walker, et al., 2007; Sun, et al., 2016 • Chit-chat systems (social bot) • Hold social conversations (Banchs, et al., 2012) • Still a very hard problem…
  10. 10. 10 Live Note/QA: http://tinyurl.com/Evorus Existing Approaches to Open Conversation • Combining multiple task-oriented dialog systems • DialPort (Zhao, et al., 2016) • End-to-end framework for dialogue systems • Serban, et al. 2016; Li, et al. 2017 • Adapting a model to many other domains • Walker, et al., 2007; Sun, et al., 2016 • Chit-chat systems (social bot) • Hold social conversations (Banchs, et al., 2012) • Still a very hard problem… MIT Technology Review Feb 27, 2018
  11. 11. 11 Live Note/QA: http://tinyurl.com/Evorus A Top-Down Approach Fully-Automated System Human- Powered System
  12. 12. 12 Live Note/QA: http://tinyurl.com/Evorus A Top-Down Approach Fully-Automated System Crowd-AI System Human- Powered System
  13. 13. 13 Live Note/QA: http://tinyurl.com/Evorus A Top-Down Approach Fully-Automated System Crowd-AI System Human- Powered System
  14. 14. 14 Live Note/QA: http://tinyurl.com/Evorus In Today’s Talk… Fully-Automated System Evorus Chorus
  15. 15. 15 Live Note/QA: http://tinyurl.com/Evorus Chorus: A Crowd-Powered Conversation Assistant Evorus
  16. 16. 16 Live Note/QA: http://tinyurl.com/Evorus •Crowd workers collectively hold a conversation by: 1. Propose Responses 2. Vote Responses 3. Take Notes Reward points for each action "Is there anything else I can help you with?": Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. Huang, et al. HCOMP 2016. Chorus: A Crowd-Powered Conversation Assistant
  17. 17. 17 Live Note/QA: http://tinyurl.com/Evorus User Interface 17 / 31
  18. 18. 18 Live Note/QA: http://tinyurl.com/Evorus User & Worker Interface 18 / 31
  19. 19. 19 Live Note/QA: http://tinyurl.com/Evorus
  20. 20. 20 Live Note/QA: http://tinyurl.com/Evorus We Deployed Chorus • Launched on May 20th, 2016 • On Google Hangouts • 2200+ conversations, 420+ users • TalkingToTheCrowd.org
  21. 21. 21 Live Note/QA: http://tinyurl.com/Evorus Fully-Automated System Evorus Chorus Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time
  22. 22. 22 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus
  23. 23. 23 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus
  24. 24. 24 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus
  25. 25. 25 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus
  26. 26. 26 Live Note/QA: http://tinyurl.com/Evorus Chatbots How to select chatbots automatically?
  27. 27. 27 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot
  28. 28. 28 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot ~= Overall Message Acceptance Rate
  29. 29. 29 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Topic Similarity User Message Domain of the Chatbot Hey what should I eat in Montreal?
  30. 30. 30 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Topic Similarity User Message Domain of the Chatbot Hey what should I eat in Montreal? Find me some good restaurants ! Where can I get Chinese food? Example Triggering Message
  31. 31. 31 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Topic Similarity User Message Domain of the Chatbot Hey what should I eat in Montreal? Example Triggering Message Find me some good restaurants ! Where can I get Chinese food? Topic Similarity
  32. 32. 32 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot
  33. 33. 33 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot  Add more chatbots over time !
  34. 34. 34 Live Note/QA: http://tinyurl.com/Evorus
  35. 35. 35 Live Note/QA: http://tinyurl.com/Evorus Automatic Upvote How to estimate the impact of an automation?
  36. 36. 36 Live Note/QA: http://tinyurl.com/Evorus Find the Best Confidence Threshold • High Threshold • Only vote when pretty sure • High precision, but little benefit • Low Threshold • Nearly always vote • Grant agreement bonus by mistake • Damage conversation quality
  37. 37. 37 Live Note/QA: http://tinyurl.com/Evorus Find the Best Confidence Threshold Expected Reward Points Saved
  38. 38. 38 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus Over Time
  39. 39. 39 Live Note/QA: http://tinyurl.com/Evorus Automating Open Conversations • Setup • A 5-month-long deployment, 80 Users • 4 chatbots + 1 voting bot • Result • Automated responses were chosen 12.44% of the time. • Human upvotes were reduced by 13.81%. • The cost of each message is reduced by 32.76%. • Conversation quality and user satisfaction level remains. • Conversation Quality: Satisfaction, Clarity, Responsiveness, Comfort (Liu, et al., 2010)
  40. 40. 40 Live Note/QA: http://tinyurl.com/Evorus Conclusion & Future Work • Automating some part of an open conversation is possible. • Future Work • 1000+ Chatbots! • Deploy Chorus as an Open Research Platform • Chorus on Smart Devices
  41. 41. 41 Live Note/QA: http://tinyurl.com/Evorus Conclusion & Future Work • Automating some part of an open conversation is possible. • Future Work • 1000+ Chatbots! • Deploy Chorus as an Open Research Platform • Chorus on Smart Devices @windx0303 KennethHuang.cc Ting-Hao (Kenneth) Huang QA: http://tinyurl.com/Evorus
  42. 42. 42 Live Note/QA: http://tinyurl.com/Evorus Backup Slides
  43. 43. 43 Live Note/QA: http://tinyurl.com/Evorus
  44. 44. 44 Live Note/QA: http://tinyurl.com/Evorus Features of Automatic Voting

×