Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

An Approach for Ex-Post-Facto Analysis of Knowledge Graph-driven Chatbots - the DBpedia Chatbot

9 views

Published on

These slides were presented at the CONVERSATIONS 2019 - 3rd International Workshop on Chatbot Research. The preprint of the corresponding paper can be found here: https://conversations2019.files.wordpress.com/2019/11/conversations_2019_paper_22_preprint.pdf

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

An Approach for Ex-Post-Facto Analysis of Knowledge Graph-driven Chatbots - the DBpedia Chatbot

  1. 1. An Approach for Ex-Post-Facto Analysis of Knowledge Graph-Driven Chatbots – the DBpedia Chatbot Rricha Jalota, Priyansh Trivedi, Gaurav Maheshwari, Axel-Cyrille Ngonga Ngomo, Ricardo Usbeck November 20, 2019 Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 1 / 17
  2. 2. Introduction Figure: U.S. Chatbot Market by Vertical, 2014 - 2025 (USD Million) 1 1 Source: https://www.grandviewresearch.com/industry-analysis/chatbot-market Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 2 / 17
  3. 3. Introduction Knowledge Graphs and Knowledge Graph-Driven Systems Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 3 / 17
  4. 4. Background Knowledge Graph-Driven Chatbot: The DBpedia Chatbot Deployed in August 2017 Purpose2 - Answer factual questions - Answer questions related to DBpedia - Expose the research work being done in DBpedia as product features - Casual conversation/banter 2 Source: https://wiki.dbpedia.org/blog/meet-dbpedia-chatbot Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 4 / 17
  5. 5. Background Knowledge Graph-Driven Chatbot: The DBpedia Chatbot Deployed in August 2017 Purpose2 - Answer factual questions - Answer questions related to DBpedia - Expose the research work being done in DBpedia as product features - Casual conversation/banter Hybrid Chatbot - domain-specific information (DBpedia-centric FAQs) + domain-agnostic factual questions (using DBpedia KG) 2 Source: https://wiki.dbpedia.org/blog/meet-dbpedia-chatbot Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 4 / 17
  6. 6. Background Case Study: The DBpedia Chatbot Total: 9084 users, 90,800 interactions Table: Feedback Statistics Feedback-asked 28953 Feedback-received 7561 Negative-feedback 4155 Figure: Architecture of the DBpedia Chatbot Check http://chat.dbpedia.org https://github.com/dbpedia/chatbot Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 5 / 17
  7. 7. Objective of the Ex-Post-Facto Analysis Understand the nature of user-requests - query-patterns - user-intentions Examine whether the chatbot can serve its purpose – satisfy user-requests Get insights about the conversation flow to improve the chatbot’s architecture Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 6 / 17
  8. 8. Approach Overview Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 7 / 17
  9. 9. Approach Request Analysis - Intent Analysis Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 8 / 17
  10. 10. Approach Request Analysis - Intent Analysis Figure: Visualization of clusters obtained via HDBSCAN on sentence embeddings. Each cluster consists of at least 25 samples. The top 10 clusters out of a total of 33 have been labeled with their top terms. Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 9 / 17
  11. 11. Approach Request Analysis - Complexity of utterances Complex Query Example: Can you give me the names of women born in the Country during the 19th century? Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 10 / 17
  12. 12. Approach Request Analysis - Miscellaneous Analysis Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 11 / 17
  13. 13. Response Analysis Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 12 / 17
  14. 14. Response Analysis Entity Types in Utterances prior to Negative Feedback Figure: Entity type distribution from 1000 manually annotated failed utterances. Table: spaCy-NER and DBpedia Spotlight accuracy for detecting person and location mentions. System Person Location spaCy-NER 41.3% 42.2% DBpedia Spotlight 69.2% 46.1% Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 13 / 17
  15. 15. Conversation Analysis Figure: Topics as identified by DBpedia Spotlight Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 14 / 17
  16. 16. Implications for DBpedia Chatbot Adding support for multilingualism Smart Suggestions Detecting implicit feedback and out-of-scope queries Knowledge-based QA Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 15 / 17
  17. 17. Implications for Knowledge-driven Chatbots Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 16 / 17
  18. 18. That’s all Folks! Get in touch: Rricha Jalota Data Science Group, Paderborn University rricha.jalota@uni-paderborn.de github.com/dice-group/DBpedia-Chatlog-Analysis Follow us on Twitter: @DiceUPB, @FraunhoferIAIS, @RrichaJalota Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 17 / 17

×