1.
Battling Model Decay with
Deep Learning and
Gamification
Sinan Ozdemir
Director of Data Science @ Directly

2.
Machine learning models
degrade over time without
continuous up-keep

3.
Up-keep is expensive, time
consuming, and often a
reactive process

4.
Directly proactively and continuously
enhances virtual agents by leveraging
community experts to boost automation
rates and help fellow customers.

5.
Agenda
Trending Cluster Analysis
Using deep learning + clustering to identify trending
intents for automation opportunities
24/7 Intent Discovery and Training
Validating intents with in-house automation
specialists and distributing intent labelling tasks with
a gamified training platform
Predictive Routing - Deep learning
Triage
Using explicit and implicit signals to continuously
train models to optimize business KPIs

6.
Trending Cluster Analysis

7.
Daily cluster analyses are used to
understand the voice of our end-users

8.
Nightly Clustering to Discover Trending Intents

9.
Nightly Clustering to Discover Trending Intents
High-Dimensional
Clustering
Question texts
extracted
Clusters are analyzed by
in-house Automation Ops and
training is sent to Experts
1) Pre and
post-sales tool
2) Intent
Discovery for
Automation
I need help
with…..
Our Understand Product

10.
What this
looks like
Global
Travel &
Hospitality
Company
3/1 - 3/15
Location Specific
COVID Refunds
Disney
Canada
Edit Profile Issue
Israel

11.
Cluster
drift in two
weeks
Global
Travel &
Hospitality
Company
3/16 - 3/31
COVID Refunds
Canada
Edit Profile Issue
COVID Refund
Status
Credit Card Charged
When will refund come?
Only received partial refund

12.
24/7 Intent Discovery and
Training

13.
Trending Clusters are potential intents to
automate

14.
Intent Discovery and Training with Gamified Platform
Trending Clusters are sent to
Automation Ops to validate
Question texts
extracted
Expert labelled data are used
to re-train intent classifiers
I need help
with…..
Models are
served in
production for
automation

15.
Content Slides
Continuous Intent
Training with experts
turned on
What that looks like
Global Consumer Electronics Company saw 2.5x rise in automation
attempts within days

16.
Predictive Routing
Deep learning based triage

17.
Using deep learning, we can effectively
route/block questions to our AI/Expert
network

18.
Nightly Predictive Routing Retraining Pipeline
Questions and Expert
behavior are extracted for
analysis
Predictive Routing learns
from implicit Expert
behavior and explicit
intent predictions
Predictive
Routing
models are
updated
I need help
with…..
Predictive
Routing blocks
or accepts
question into
Expert network

19.
What that looks like
Large Gaming Company Case Study

20.
Service
Outage
Catches new topic
and recovers
Fails harder
PredictivePower(weightedF1)

21.
Summary
▪ Surface potential
intents for automation
opportunities
▪ Pre and post-sales
tool to help customers
understand their data
▪ Automation
specialists validate
new intents. Experts
further train models
▪ Increase deflection
of customer
inquiries
24/7 Intent DiscoveryTrending Cluster Analysis
▪ Effectively route
questions by
learning from Expert
behavior
▪ Optimize Resolution
rate, CSAT, etc
Predictive Routing Retraining

22.
Battling Model Decay with
Deep Learning and
Gamification
Sinan Ozdemir
Director of Data Science @ Directly

23.
Q/A

24.
Feedback
Your feedback is important to us.
Don’t forget to rate and
review the sessions.