Abstract:- Sophisticated machine learning models (like GBMs and Neural Networks) produce better predictions than simpler models (like linear or logistic regression), but sophisticated models do not produce interpretable 'effects' that specify the relationship between predictors and and outcome. This is because sophisticated models can learn non-linear, interactive, or even higher level relationships between the predictors and outcome without being explicitly specified. In many settings it is often important to understand, as best as possible, how 'black box' models are producing because:1. If users do not understand how a prediction is being made, they may not trust the model/prediction enough to act upon the model's suggestions. Significant business value can be derived from understanding what drives an outcome of interest (e.g. purchase or churn) in order to make product changes to accentuate or minimize desired effects 3. Understanding how predictors relate to an outcome can inform subsequent feature generation that can improve a model's predictive power. This talk will discuss two methods that have been proposed to better understand machine learning models: simulating changes in input variables (the R ICEbox package) or building a simpler model locally around specific predictions (the Python LIME package).

1. Opening the black box: Attempts to understand
the results of machine learning models
Michael Tiernay, PhD
R&D Data Scientist, Edmunds.com
07/29/2017

2. The Problem

4. Inferential Models Might Be Bad
−300
0
300
600
0 100 200 300
temperature
pressure

5. Non-Parametric Models Are Hard to Understand

8. Local Vs. Global Interpretability

9. Local Vs. Global Interpretability
1. Local Interpretability - Focus on how a model works around a
single or cluster of simililar observations
2. Global Interpretability - Focus on how a model works across all
observations (i.e. coefficients from a liner regression)

10. Why Do we Want Local Interpretability?
Undrestand why a prediction is positive/negative
Trust individual predictions (i.e. reasons for a prediction make
sense to domain experts)
Provide guidence for intervening strategies (i.e. the cancer is
predicted to be caused by X, which can be treated with Y)
These problems have been addressed by recent literature

11. Why Do we Want Global Interpretability?
Hypothesis Generation: Model can help generate new ideas
that can be tested experimentally
A global understanding of the ‘causes’ of an outcome can drive
significant business/product changes
This problem has not received much attention in the machine
learning literature

12. Let’s Look At a Toy Example

13. Logistic Regression Example

14. Raw Data
## Survived Pclass Sex Age
## 1 class_0 3 2 22
## 2 class_1 1 1 38
## 3 class_1 3 1 26
## 4 class_1 1 1 35
## 5 class_0 3 2 35
## 7 class_0 1 2 54

15. Logistic Regression
##
## Call:
## NULL
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -2.7270 -0.6799 -0.3947 0.6483 2.4668
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 7.578137 0.618091 12.261 < 2e-16 ***
## Pclass -1.288545 0.139259 -9.253 < 2e-16 ***
## Sex -2.522131 0.207283 -12.168 < 2e-16 ***
## Age -0.036929 0.007628 -4.841 1.29e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1
##

16. Lime (Locally Interpretable Model-agnostic
Explanations)

17. Lime (Locally Interpretable Model-agnostic Explanations)
Mainly created for images and text
Model agnostic
Focus on one observation (x) at a time
Sample other observations (z) weighted by distance to x
Compute f(z) (The predicted outcome)
Select K features with LASSO then compute least squares
Coefficients from LS are ‘local effects’

18. Local Interpretability Example

19. Titanic Effects (Men)

20. Titanic Effects (Women)

21. Lime’s Conclusions
1. Women survive at a higher rate than men
2. Being in third class has a substantially more negative effect
than being in second class for both men and women
3. The effect of being in third class and second class is the same
for men and women
4. Age has small positive effects on men and small negative
effects on women

22. Simple Simulations

23. Simulate Changes By Gender
Change Female to Male Change Male to Female
−0.4 0.0 0.4 −0.4 0.0 0.4
0
25
50
75
100
Change In Survival Probability
count
colour
red

24. Simulate Changes By Class
Class 1 to 3 Class 2 to 3 Class 3 to 2
Class 1 to 2 Class 2 to 1 Class 3 to 1
−0.4 0.0 0.4 −0.4 0.0 0.4 −0.4 0.0 0.4
0
50
100
150
200
0
50
100
150
200
Change In Survival Probability
count
colour
red

25. By Class (and Gender)
Class 1 to 3 Class 2 to 3 Class 3 to 2
Class 1 to 2 Class 2 to 1 Class 3 to 1
−0.4 0.0 0.4 −0.4 0.0 0.4 −0.4 0.0 0.4
0
50
100
150
200
0
50
100
150
200
Change In Survival Probability
count
sex
Female
Male
colour
red

26. Same Passengers as Lime
0.2
0.4
0.6
0.8
Class 1 Class 2 Class3
Passenger Class
LikelihoodofSurvival
Sex
Female
Male

27. Simulate A 1 Year Change in Age
Negative Positive
−0.1 0.0 0.1 −0.1 0.0 0.1
0
100
200
300
400
500
Change In Survival Probability
NumberofPassengers

28. Combine Both Effects
0
100
200
300
400
−0.05 0.00 0.05 0.10 0.15
Change In Survival Probability
NumberofPassengers

29. Plot Individual Effect By Age
−0.05
0.00
0.05
0.10
0.15
0 20 40 60 80
Passenger Age
IndividualEffectofChangingAgeby1

30. Surivial by Age for All Passengers
0.2
0.4
0.6
0.8
0 20 40 60 80
Age
ProbabilityofSurvival
group
Female−1st
Female−2nd
Female−3rd
Male−1st
Male−2nd
Male−3rd

31. Ice Box - Gender
−10123
Sex
partiallog−odds
0 1

32. Thoughts

33. Shortcomings of Lime
Good out-of-the-box solution that requires little thought
Doesn’t allow control over local space
Need to center/scale features for distance calculation
Different effects up and down of binary/ordinal features

34. My Simulation Ideas
Simple simulations reveal a lot
Calculating effects for binary predictors is trivial
Calculating effects for categorical predictors is harder (with
many categories)
Numeric Predictors:
Examine the effect of a change of a specific size across the
entire population (Similar to average partial effects in
econometrics)
Look at one observation and see how predictions change across
the universe of the one predictor
Simulate every possible value of a predictor for every
observation (or a random subset)

35. Links

36. https:
//datascienceinc.github.io/Skater/overview.html
https://www.oreilly.com/ideas/
ideas-on-interpreting-machine-learning
https://github.com/marcotcr/lime
https://www.oreilly.com/learning/
introduction-to-local-interpretable-model-agnostic-expl
http:
//marcotcr.github.io/lime/tutorials/Tutorial%20-%
20continuous%20and%20categorical%20features.html
https://arxiv.org/pdf/1602.04938.pdf

37. http://scweiss.blogspot.com/2015/12/
beyond-beta-relationships-between.html
https://cran.r-project.org/web/packages/ICEbox/
ICEbox.pdf
https://arxiv.org/pdf/1309.6392.pdf
https://stats.stackexchange.com/questions/73449/
average-partial-effects