Successfully reported this slideshow.

Battling Model Decay with Deep Learning and Gamification



1 of 25
1 of 25

More Related Content

Similar to Battling Model Decay with Deep Learning and Gamification

Battling Model Decay with Deep Learning and Gamification

  1. 1. Battling Model Decay with Deep Learning and Gamification Sinan Ozdemir Director of Data Science @ Directly
  2. 2. Machine learning models degrade over time without continuous up-keep
  3. 3. Up-keep is expensive, time consuming, and often a reactive process
  4. 4. Directly proactively and continuously enhances virtual agents by leveraging community experts to boost automation rates and help fellow customers.
  5. 5. Agenda Trending Cluster Analysis Using deep learning + clustering to identify trending intents for automation opportunities 24/7 Intent Discovery and Training Validating intents with in-house automation specialists and distributing intent labelling tasks with a gamified training platform Predictive Routing - Deep learning Triage Using explicit and implicit signals to continuously train models to optimize business KPIs
  6. 6. Trending Cluster Analysis
  7. 7. Daily cluster analyses are used to understand the voice of our end-users
  8. 8. Nightly Clustering to Discover Trending Intents
  9. 9. Nightly Clustering to Discover Trending Intents High-Dimensional Clustering Question texts extracted Clusters are analyzed by in-house Automation Ops and training is sent to Experts 1) Pre and post-sales tool 2) Intent Discovery for Automation I need help with….. Our Understand Product
  10. 10. What this looks like Global Travel & Hospitality Company 3/1 - 3/15 Location Specific COVID Refunds Disney Canada Edit Profile Issue Israel
  11. 11. Cluster drift in two weeks Global Travel & Hospitality Company 3/16 - 3/31 COVID Refunds Canada Edit Profile Issue COVID Refund Status Credit Card Charged When will refund come? Only received partial refund
  12. 12. 24/7 Intent Discovery and Training
  13. 13. Trending Clusters are potential intents to automate
  14. 14. Intent Discovery and Training with Gamified Platform Trending Clusters are sent to Automation Ops to validate Question texts extracted Expert labelled data are used to re-train intent classifiers I need help with….. Models are served in production for automation
  15. 15. Content Slides Continuous Intent Training with experts turned on What that looks like Global Consumer Electronics Company saw 2.5x rise in automation attempts within days
  16. 16. Predictive Routing Deep learning based triage
  17. 17. Using deep learning, we can effectively route/block questions to our AI/Expert network
  18. 18. Nightly Predictive Routing Retraining Pipeline Questions and Expert behavior are extracted for analysis Predictive Routing learns from implicit Expert behavior and explicit intent predictions Predictive Routing models are updated I need help with….. Predictive Routing blocks or accepts question into Expert network
  19. 19. What that looks like Large Gaming Company Case Study
  20. 20. Service Outage Catches new topic and recovers Fails harder PredictivePower(weightedF1)
  21. 21. Summary ▪ Surface potential intents for automation opportunities ▪ Pre and post-sales tool to help customers understand their data ▪ Automation specialists validate new intents. Experts further train models ▪ Increase deflection of customer inquiries 24/7 Intent DiscoveryTrending Cluster Analysis ▪ Effectively route questions by learning from Expert behavior ▪ Optimize Resolution rate, CSAT, etc Predictive Routing Retraining
  22. 22. Battling Model Decay with Deep Learning and Gamification Sinan Ozdemir Director of Data Science @ Directly
  23. 23. Q/A
  24. 24. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.