Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
1 of 33

The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and NLP to Measure Customer Service Quality



Download to read offline

How do we get better than good enough? Leveraging NLP techniques, we can determine the general sentiment of a sentence, phrase, or a paragraph of text. We can mine the world of social data to get a sense of what is being said. But, how do you get control of the factors that create happiness? How do you become proactive in making end-users happy? Chatbots, human chats, and conversations are the means we are using to express our ideas to each other. NLP is great for helping us process and understand this data but can fall short. In our session, we will explore how to expand NLP/sentiment analysis to investigate the intense interactions that can occur between humans and humans or humans and robots. We will show how to pinpoint the things that work to improve quality and how to use those data points to measure the effectiveness of chatbots. Learn how we have applied popular NLP frameworks such as NLTK, Stanford CoreNLP and John Snow Labs NLP to financial customer service data. Explore techniques to analyze conversations for actionable insights. Leave with an understanding of how to influence your customers' happiness.

The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and NLP to Measure Customer Service Quality

  1. 1. WIFI SSID:SparkAISummit | Password: UnifiedAnalytics
  2. 2. Jaime Woodfin, FIS Global Brooke Wenig, Databricks Special thanks to Amir Issaei, Kevin Mellott, and Aaron Colcord #UnifiedAnalytics #SparkAISummit
  3. 3. Outline • FIS & Databricks Intro • Business Problem & Motivation • Approach v1 – Problems encountered along the way • Approach v2 • Production 3#UnifiedAnalytics #SparkAISummit
  4. 4. Who is FIS Global? • The Global leader in financial services technology • Customers: banks and credit unions • Ecosystem of products and services built around core banking • FIS Digital Finance, Digital Data and Analytics 4#UnifiedAnalytics #SparkAISummit
  5. 5. Accelerate innovation by unifying data science, engineering and business • Founded by the original creators of Apache Spark • Contributes 75% of the open source code, 10x more than any other company • Trained 100k+ Spark users on the Databricks platform VISION WHO WE ARE Unified Analytics Platform powered by Apache SparkPRODUCT
  6. 6. Business Problem
  7. 7. Conversational Analytics • Measure in support conversations • Follow-up on support conversation 7#UnifiedAnalytics #SparkAISummit
  8. 8. Example Conversation
  9. 9. Conversational Channels Developments 9#UnifiedAnalytics #SparkAISummit Face-to-face Human support chat Support chatbots Conversational Banking
  10. 10. Goals • Score overall user conversation satisfaction • Question: What contributed to their satisfaction? • AND how to do it at scale?? 10#UnifiedAnalytics #SparkAISummit
  11. 11. Approach v1
  12. 12. Approach v1 • Apply open-source NLP libraries to each turn in conversation 12#UnifiedAnalytics #SparkAISummit
  13. 13. Library Comparison • Different Scales – TextBlob: [-1, 1] – NLTK: [-1, 1] – John Snow Labs (sparknlp): Negative or Positive – Stanford CoreNLP: 0, 1, 2, 3, 4 13#UnifiedAnalytics #SparkAISummit
  14. 14. Demo
  15. 15. Problems Encountered • Stanford CoreNLP gave predictions per sentence, not turn • Performed poorly on neutral sentences – John Snow Labs had no “neutral” category • Didn’t do well with banking domain Can we do better? 15#UnifiedAnalytics #SparkAISummit
  16. 16. Approach v2
  17. 17. Approach v2 • No pre-trained sentiment analysis models! • Model: – Build LSTM model on all conversation text to predict sentiment – Augment with additional features (e.g. # of turns, time of day, etc.) – Pass features through end classifier • Positive/Negative 17#UnifiedAnalytics #SparkAISummit X
  18. 18. Transfer Learning • Distributed training of LSTM on open-source sentiment dataset with HorovodRunner • Transfer learning on banking data 18#UnifiedAnalytics #SparkAISummit
  19. 19. LSTM Stats • Basic stats for performance: – Accuracy: 73% – FPR: 11% – FNR: 32% 19#UnifiedAnalytics #SparkAISummit 𝐹𝑃𝑅 = 𝐹𝑃 𝑁 = 𝐹𝑃 𝐹𝑃 + 𝑇𝑁 𝐹𝑁𝑅 = 𝐹𝑁 𝑁 = 𝐹𝑁 𝑇𝑃 + 𝐹𝑁
  20. 20. Features • LSTM output • Conversational Features – User average turn length – Agent average turn length – # Turns – Duration • Temporal Features – Day of Week – Time of Day • Others 20#UnifiedAnalytics #SparkAISummit
  21. 21. Classifier • Logistic Regression – Accuracy: 77% – FPR: 24% – FNR: 22% • Random Forests – Accuracy: 80% – FPR: 10% – FNR: 51% • Others 21#UnifiedAnalytics #SparkAISummit
  22. 22. Random Forest • Chose the Random Forest b/c: – Lowest FP rate & Highest Accuracy – Good model interpretability – Part of SparkML and can use with Pipeline API (easy to switch to Scala) 22#UnifiedAnalytics #SparkAISummit
  23. 23. Production
  24. 24. Production Requirements • Fit into dev pipeline, largely Scala/Java based – But a lot of data science is done in Python • Close the feedback loop - constantly learning • Automated deployments • Streaming instead of batch 24#UnifiedAnalytics #SparkAISummit
  25. 25. Architecture 25#UnifiedAnalytics #SparkAISummit
  26. 26. Architecture 26#UnifiedAnalytics #SparkAISummit
  27. 27. Python & Scala • Train LSTM in Python (Keras) • Save Model • Load model via UDF • Apply using Scala! 27#UnifiedAnalytics #SparkAISummit
  28. 28. Deployment 28#UnifiedAnalytics #SparkAISummit
  29. 29. Recap 29#UnifiedAnalytics #SparkAISummit Idea Notebook Production Happy
  30. 30. Thank you!
  31. 31. Life is but a Stream @ 3:30 pm Thursday Aaron Colcord & Kevin Mellott (FIS Global) Streaming Track, ROOM 2007
  32. 32. Questions?