This session focuses on how AI can be used to enhance a company’s CRM using a case study where RIT Singapore closely collaborated with Rakuten Viki to implement this. It will cover topics such as customer churn, acquisition and segmentation. It will cover both successes and failures that were encountered during the process and will be useful not just for machine learning practitioners but also for people working in marketing, finance and strategy who are interested in integrating machine learning in their work.
2. 2
I have built a model to predict which customer will cancel their subscription
..why is marketing not using it?
1. Marketing team will not use the model if they don’t know what action to take
on the predictions and why
2. Data Scientists can use model interpretation to help marketing team take
action
6. 6
…
Yes, sure.
That sounds
exciting
Management
Why don't we
start using AI to
identify which
customers will
churn next
month ?
Marketing
Can you help
identify customers
who will churn next
month?
Data Scientist
7. 7
…
Sure. I can use
past data to build
a model to predict
who will churn in
the next month.
Data Scientist
10. 10
…
If client is unable to action on predictions then the model will not be used
How should the output of the
model be used ? Given the
high risk churners what should
I do and why?
Marketing
12. 12
Customer Probability to churn Reason for prediction
X 0.27 Used service for more than year, Actively watching content
Y 0.86 New User, Cancelled in the past
Z 0.82
Used service for more than year, Lost Interest
CHURN MODEL PREDICTIONS EXPLANATION
13. 13
MODEL INTERPRETATION
Model Interpretation of a machine learning model is how humans can
understand the choices made by the model in the decision making process
MODEL PREDICTIONS EXPLANATION
15. 15
*(see the SHAP NIPS paper for details).
SHAP*:
• Latest work on model interpretation
• Explains the output of any machine learning model.
• Connects game theory with model explanations
• Unifies several previous methods (such as LIME, DeepLift)
16. 16
Member A
What is an individual’s contribution in the team?
Team makes $400 profits
Member B Member C
17. 17
Mapping teams to model prediction (Example)
Team Members Features used for a prediction
Team sales Prediction Score
Member A
Member A
Member A
$400 0.78
Days since subscription
Minutes Viewed
Number of past subscriptions
18. 18
1: Calculate Normalizing Weights for all sub-sizes of teams
Team Size Normalizing Weight* Team Permutations
1
2
3
33%
16%
33% *Normalizing Weight = m! x (n-m-1)! / n!
n = size of original team
m = size of team player get’s added to
19. 19
2: Contribution when each member is working alone
$200 $100 $0
Member A Member B Member C
20. 20
3: Contribution of players as part of teams
B’s contribution
(in a team with A)
= 400 - 200 = $200
A’s contribution
(in a team with B)
= 400 - 100 = $300
+ = $400
Member A Member B
21. 21
4: Final contributions
Final SHAP AttributionsNormalizing Weights
Team of 1
$200
$100
$0
33% 16% 33%
$200 + $300
= $500
=$300
$0
$300
$200
$0
Team of 2 Team of 3
33% x $200 + 16% x $500 + 33% x $300
= $250
= $0
= $150
Member A
Member B
Member C
22. 22
No. of past cancellations
= 5
Prediction with a high churn score
23. 23
Explanation of the prediction of the high churn score
days since subscription = 25
Minutes viewed last
quarter
(of subscription) = 0
No. of past cancellations = 5
24. 24
Days since subscription = 311
Minutes viewed last quarter
(of subscription) = 3751
Days since last watched video = 0
Sample explanation of a low churn score
27. 27
Days since subscription = 31
days
Minutes viewed last quarter = 0
New users,
not watching content
Higher
lower
S
H
A
P
V
A
L
U
E
S
USERS
Helps marketing understand the decisions of the model
29. 29
Marketing Planning a campaign for disengaged users using content recommendation
Acquisition Date: 24th Sep 2018
30. 30
Revisit model / data
Higher
lower
Minutes viewed in 2nd quarter of subscription =
1000
Minutes viewed drop > 50 %
Minutes viewed in last quarter = 0
S
H
A
P
V
A
L
U
E
S
Helps in in Qualitative Model Evaluation
31. 31
• Churn prediction is just one component of churn management
• Marketing will not use the model if they don’t know what action to take on the
predictions and why
• Use model interpretation to help marketing take action