Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time
Report
Ting-Hao HuangFollow
Apr. 26, 2018•0 likes•947 views
Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time
Apr. 26, 2018•0 likes•947 views
Download to read offline
Report
Technology
Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time. Ting-Hao K. Huang, Joseph Chee Chang, Jeffrey P. Bigham. In Proceedings of Conference on Human Factors in Computing Systems 2018 (CHI 2018), 2018, Montréal, Canada.
Ting-Hao HuangFollow
Recommended
A Crowd-Powered Conversational Assistant That Automates Itself Over TimeTing-Hao Huang
DEF CON 23 - Rich Kelley - harness powershell weaponization made easyFelipe Prado
Deployments in one click!Manuel de la Peña Peña
How to "Hack" the DSpace CommunityTim Donohue
Revolutionize Your Workflow with ChatOpsTessa Mero
Bringing Dev and Ops together with ChatOpsJaap Brasser
More Related Content
Similar to Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time
Using Chatbots in Extension ProgrammingAmy Cole
Measure camp tools of the cro rabbleCraig Sullivan
Automating web application security testing using Hubot AI / NLP chat bot - B...Abhijith B R
A Case Study of Using Selenium IDE and WebDriver_PresenationJabeen Shazia Posses H1 B Visa (Jazz)
Open source-secret-sauce-rit-2010Ted Husted
Abraham aranguren. legal and efficient web app testing without permissionYury Chemerkin
More from Ting-Hao Huang
A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crow...Ting-Hao Huang
Real-time On-Demand Crowd-powered Entity ExtractionTing-Hao Huang
A Crowd-Powered Conversational Assistant That Automates Itself Over TimeTing-Hao Huang
"Is there anything else I can help you with?": Challenges in Deploying an On-...Ting-Hao Huang
Visual Storytelling (NAACL 2016, Poster)Ting-Hao Huang
Social Metaphor Detection via Topical AnalysisTing-Hao Huang
Recently uploaded
Advancing Equity and Inclusion for Deaf Students in Higher Education3Play Media
Machine learning with quantum computersSpeck&Tech
LLaMA 2.pptxRkRahul16
Netwitness RT - Don’t scratch that patch.pptxStefano Maccaglia
Document Understanding as Cloud APIs and Generative AI Pre-labeling Extractio...DianaGray10
info_session_gdsc_tmsl .pptxNikitaSingh741518
Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time
- 1. 1 Live Note/QA: http://tinyurl.com/Evorus Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time Ting-Hao (Kenneth) Huang* Joseph Chee Chang Jeffrey P. Bigham [ Question / Feedback: http://tinyurl.com/Evorus ] *Kenneth will join in Fall 2018
- 7. 7 Live Note/QA: http://tinyurl.com/Evorus What just happened? • Open Conversation • Multi-turn interaction • Multiple domains • Personalized • Coherent dialog • Mix of task-oriented and social conversation
- 8. 8 Live Note/QA: http://tinyurl.com/Evorus Today’s Conversational Assistants… “What’s new with Alexa?”“Talking to Siri”
- 9. 9 Live Note/QA: http://tinyurl.com/Evorus Existing Approaches to Open Conversation • Combining multiple task-oriented dialog systems • DialPort (Zhao, et al., 2016) • End-to-end framework for dialogue systems • Serban, et al. 2016; Li, et al. 2017 • Adapting a model to many other domains • Walker, et al., 2007; Sun, et al., 2016 • Chit-chat systems (social bot) • Hold social conversations (Banchs, et al., 2012) • Still a very hard problem…
- 10. 10 Live Note/QA: http://tinyurl.com/Evorus Existing Approaches to Open Conversation • Combining multiple task-oriented dialog systems • DialPort (Zhao, et al., 2016) • End-to-end framework for dialogue systems • Serban, et al. 2016; Li, et al. 2017 • Adapting a model to many other domains • Walker, et al., 2007; Sun, et al., 2016 • Chit-chat systems (social bot) • Hold social conversations (Banchs, et al., 2012) • Still a very hard problem… MIT Technology Review Feb 27, 2018
- 11. 11 Live Note/QA: http://tinyurl.com/Evorus A Top-Down Approach Fully-Automated System Human- Powered System
- 12. 12 Live Note/QA: http://tinyurl.com/Evorus A Top-Down Approach Fully-Automated System Crowd-AI System Human- Powered System
- 13. 13 Live Note/QA: http://tinyurl.com/Evorus A Top-Down Approach Fully-Automated System Crowd-AI System Human- Powered System
- 14. 14 Live Note/QA: http://tinyurl.com/Evorus In Today’s Talk… Fully-Automated System Evorus Chorus
- 15. 15 Live Note/QA: http://tinyurl.com/Evorus Chorus: A Crowd-Powered Conversation Assistant Evorus
- 16. 16 Live Note/QA: http://tinyurl.com/Evorus •Crowd workers collectively hold a conversation by: 1. Propose Responses 2. Vote Responses 3. Take Notes Reward points for each action "Is there anything else I can help you with?": Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. Huang, et al. HCOMP 2016. Chorus: A Crowd-Powered Conversation Assistant
- 17. 17 Live Note/QA: http://tinyurl.com/Evorus User Interface 17 / 31
- 18. 18 Live Note/QA: http://tinyurl.com/Evorus User & Worker Interface 18 / 31
- 20. 20 Live Note/QA: http://tinyurl.com/Evorus We Deployed Chorus • Launched on May 20th, 2016 • On Google Hangouts • 2200+ conversations, 420+ users • TalkingToTheCrowd.org
- 21. 21 Live Note/QA: http://tinyurl.com/Evorus Fully-Automated System Evorus Chorus Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time
- 26. 26 Live Note/QA: http://tinyurl.com/Evorus Chatbots How to select chatbots automatically?
- 27. 27 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot
- 28. 28 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot ~= Overall Message Acceptance Rate
- 29. 29 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Topic Similarity User Message Domain of the Chatbot Hey what should I eat in Montreal?
- 30. 30 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Topic Similarity User Message Domain of the Chatbot Hey what should I eat in Montreal? Find me some good restaurants ! Where can I get Chinese food? Example Triggering Message
- 31. 31 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Topic Similarity User Message Domain of the Chatbot Hey what should I eat in Montreal? Example Triggering Message Find me some good restaurants ! Where can I get Chinese food? Topic Similarity
- 32. 32 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot
- 33. 33 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot Add more chatbots over time !
- 35. 35 Live Note/QA: http://tinyurl.com/Evorus Automatic Upvote How to estimate the impact of an automation?
- 36. 36 Live Note/QA: http://tinyurl.com/Evorus Find the Best Confidence Threshold • High Threshold • Only vote when pretty sure • High precision, but little benefit • Low Threshold • Nearly always vote • Grant agreement bonus by mistake • Damage conversation quality
- 37. 37 Live Note/QA: http://tinyurl.com/Evorus Find the Best Confidence Threshold Expected Reward Points Saved
- 38. 38 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus Over Time
- 39. 39 Live Note/QA: http://tinyurl.com/Evorus Automating Open Conversations • Setup • A 5-month-long deployment, 80 Users • 4 chatbots + 1 voting bot • Result • Automated responses were chosen 12.44% of the time. • Human upvotes were reduced by 13.81%. • The cost of each message is reduced by 32.76%. • Conversation quality and user satisfaction level remains. • Conversation Quality: Satisfaction, Clarity, Responsiveness, Comfort (Liu, et al., 2010)
- 40. 40 Live Note/QA: http://tinyurl.com/Evorus Conclusion & Future Work • Automating some part of an open conversation is possible. • Future Work • 1000+ Chatbots! • Deploy Chorus as an Open Research Platform • Chorus on Smart Devices
- 41. 41 Live Note/QA: http://tinyurl.com/Evorus Conclusion & Future Work • Automating some part of an open conversation is possible. • Future Work • 1000+ Chatbots! • Deploy Chorus as an Open Research Platform • Chorus on Smart Devices @windx0303 KennethHuang.cc Ting-Hao (Kenneth) Huang QA: http://tinyurl.com/Evorus
- 44. 44 Live Note/QA: http://tinyurl.com/Evorus Features of Automatic Voting