Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Artemy Malkov, CEO, Data Monsters at The AI Conference 2017

1,050 views

Published on

Artemy is a CEO of Data Monsters, a Palo Alto based research lab and consulting company. Prior to Data Monsters Artemy founded a business intelligence startup, which raised $6M of venture capital and two years later was sold to a nationwide system integrator. Artemy is an expert in computational social science, knowledge mining, chaos theory.

Why Chatbots Fail, And How To Fix Them
Chatbots look damn smart at demonstrations when presenters follow the pre-designed scripts. But chatbots fail when real users come. Real users talk in an unexpected manner, change topics and so on.

Bots still have very few success stories, with very limited number of use cases. The technology did not take off. In March 2017 Facebook recommended replacing conversational experience with a three-level menu navigation. Another leader, Amazon Alexa has only one frequent use case: “Alexa, play a song”. Everything else does not stick.

The frequency of users’ requests follows the statistical distribution with the long tail. In order to keep conversation a good chatbot should be able to understand thousands topics, not dozens. That requires huge knowledge bases.

We analyzed thousands of chatbot logs and observed a significant probability of missunderstanding that multiplies with every next phrase. 10-30% of users say something which the chatbot is not prepared and trained for. Almost every long conversation frustrates the user. Retention rate is 3-5 times lower for bots than for mobile apps, which is a disaster.

We want to discuss these problems and offer technical solutions in order to improve experience, create knowledge bases faster and build useful self-learning chatbots.

Published in: Technology
  • Be the first to comment

Artemy Malkov, CEO, Data Monsters at The AI Conference 2017

  1. 1. Facebook launches Messenger platform with chatbots Apr 2016 Bots, Messenger and the future of customer service May 2016 Why and how chatbots will dominate social media Jul 2016 Will Facebook be your next call center operator? Jul 2016 The human role in a bot-dominated future Jul 2016 The new paradigm for human-bot communication Aug 2016 THE ROARING 2016
  2. 2. Facebook’s new chatbots still need work Apr 2016 The dehumanization of Facebook Messenger May 2016 Why do chatbots suck? May 2016 The impending bot backlash Jun 2016 The hidden risks of the bot explosion Jun 2016 Bursting the chatbot bubble Jul 2016 Hard questions about bot ethics Sep 2016 As Messenger’s bots lose steam, Facebook pushes menus over chat Mar 2017 THE PHANTOM MENACE
  3. 3. Natural Language Insane variability. Long tail distributions. Scripts don’t work. NLP dimensionality reduction.
  4. 4. Question
  5. 5. Answer 1 inchSize of the Message
  6. 6. Answer 1 inchSize of the Message 3 800 000 inchesSize of the Bay Area
  7. 7. Answer 500 000 000 inchesSize of the Earth 1 inchSize of the Message 3 800 000 inchesSize of the Bay Area
  8. 8. Answer 54 800 000 000 inchesSize of the Sun 500 000 000 inchesSize of the Earth 1 inchSize of the Message 3 800 000 inchesSize of the Bay Area
  9. 9. Answer 55 870 000 000 000 000 000 000 inchesSize of the Galaxy 54 800 000 000 inchesSize of the Sun 500 000 000 inchesSize of the Earth 1 inchSize of the Message 3 800 000 inchesSize of the Bay Area
  10. 10. Answer 34 848 000 000 000 000 000 000 000 000 inchesSize of the Universe 55 870 000 000 000 000 000 000 inchesSize of the Galaxy 54 800 000 000 inchesSize of the Sun 500 000 000 inchesSize of the Earth 1 inchSize of the Message 3 800 000 inchesSize of the Bay Area
  11. 11. Answer 282 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 Combinations of 59 symbols: 26 letters + “ ” ( 27 = 2.82e+84 )59 34 848 000 000 000 000 000 000 000 000 inchesSize of the Universe 55 870 000 000 000 000 000 000 inchesSize of the Galaxy 54 800 000 000 inchesSize of the Sun 500 000 000 inchesSize of the Earth 1 inchSize of the Message 3 800 000 inchesSize of the Bay Area
  12. 12. Okay, Some Words Are More Frequent Than Others the + be + to + of + and + a + + in + that + have + I 25% of English language An iceberg is a large piece of freshwater ice that has broken off a glacier or an ice shelf and is floating freely in open water. It may subsequently become frozen into pack ice (one form of sea ice). Almost 91% of an iceberg is below the surface of the water. In this text: 19 of 51 words
  13. 13. What Is Long Tail (Zipf’s Law) F = a rank
  14. 14. What Is Long Tail (Zipf’s Law) Williams, J. R. et al. Zipf’s law holds for phrases, not words. Sci. Rep. 5, 12209; doi: 10.1038/srep12209 (2015).
  15. 15. Frequent vs Mid-Frequent Frequent Mid-Frequent
  16. 16. Why Dimensionality Matters? Buttons UI Natural Language UI 2-10 variations 100 000 000 000 000 000 variations
  17. 17. Conversational Bots: Expectation
  18. 18. Reality
  19. 19. Reality
  20. 20. Reality
  21. 21. Reducing of the Dimensionality of Natural Language An iceberg is a large piece of freshwater ice that has broken off a glacier or an ice shelf and is floating freely in open water. It may subsequently become frozen into pack ice (one form of sea ice). As it drifts into shallower waters, it may come into contact with the seabed, a process referred to as seabed gouging by ice. Almost 91% of an iceberg is below the surface of the water. Sentence splitting Tokenization Spell checking Stemming / Lemmatization Vectorization Word embedding PoS tagging Topic detection Sentiment analysis Named entity recognition Relation extraction Intent detection Natural Text Pre-processing Understanding
  22. 22. Long tails again! Business verticals. Scripts don’t work again! Human Needs and Intents
  23. 23. Dozens, Hundreds, And Thousands Of Intents
  24. 24. Different Business Verticals Have Different Distributions Heavy “head” and short “tail” Heavy and long “tail”
  25. 25. Different Business Verticals Have Different Distributions Heavy “head” and short “tail” Heavy and long “tail”
  26. 26. Question Your company has 100 000 chat conversations a month. How many intents should you build in your chatbot to cover 95% of requests?
  27. 27. Question Your company has 100 000 chat conversations a month. How many intents should you build in your chatbot to cover 95% of requests? 20-50?
  28. 28. Question Your company has 100 000 chat conversations a month. How many intents should you build in your chatbot to cover 95% of requests? 20-50? NO! 5000-7000
  29. 29. Vectorization and Clustering of Conversations
  30. 30. Oh, no! Long tails again! Business Processes
  31. 31. Even An Ideal Chatbot Is Not A Silver Bullet
  32. 32. What Does All This Mean For Business? OR Humans are expensive Let’s replace human agents with chatbots!
  33. 33. What Businesses Expect From Bots Expectation
  34. 34. What Businesses Get Reality
  35. 35. Solution?
  36. 36. 1. Start with STATISTICAL ANALYSIS of conversations with your customers! 2. Understand what distributions are typical for your business 3. Semi-supervised learning from conversations (Rules and scripts won’t work for mid/low-frequent requests) 4. Retrieval of the “Semantic layer” 5. Reinforcement learning and agent-in-the-loop How To Make It Work
  37. 37. The right way to augment customer service AND The best “Chatbot” systems are those, that learn fast and augment human agents
  38. 38. Thank you for your attention! Artemy Malkov, Ph.D. CEO, Data Monsters am@datamonsters.co bots.datamonsters.co

×