H2O Driverless AI's machine learning interpretability (MLI) module provides explainable AI capabilities. It employs techniques like surrogate models, Shapley values, and LIME to explain both Driverless AI and external models. The MLI module fits into the end-to-end Driverless AI workflow and allows for global and local explanations of model behavior and feature importance to build more human-centered, low-risk models.

1. Confidential1 Confidential1
• Explainable AI is in the news, and for good reason. Financial services
companies have cited the ability to explain AI-based decisions as one of
the critical roadblocks to further adoption of AI for their industry.
Transparency, accountability, and trustworthiness of data-driven decision
support systems based on AI and machine learning are serious regulatory
mandates in banking, insurance, healthcare, and other industries. From
pertinent regulations, to increasing customer trust, data scientists and
business decision makers must show AI-based decisions can be
explained.
• H2O Driverless AI does explainable AI today with its machine learning
interpretability (MLI) module. This capability in H2O Driverless AI employs
a unique combination of techniques and methodologies to explain the
results of both Driverless AI models and external models.
Explainable AI with H2O Driverless AI's machine
learning interpretability module

2. Explainable AI
with H2O Driverless AI’s
machine learning interpretability
module
Martin Dvorak
Software Engineer, H2O.ai
martin.dvorak@h2o.ai
H2O.ai Prague Meetup #3 2019/5/16

3. ABOUT
ME
Martin is a passionate software engineer and RESTafarian who is interested in machine
learning, VM construction, enterprise software and knowledge management. He
holds Master degree in Computer Science from Charles University Prague with
specializations in compilers, operating systems and AI/ML.
Martin is a backend engineer on the MLI project at H2O.ai

4. AGENDA
• Intro
– Context and scope.
• Why
– Explainability matters.
• What
– Steps to build human-centered, low-risk models.
• How
– Explaining models using of H2O.ai’s solution.

5. Intro
Terminology, scope and context

6. Confidential6
Terminology, Scope and Context
• Machine Learning Interpretability
– “[Machine learning interpretability] is the ability to explain or present in
understandable terms to a human.“ –https://arxiv.org/pdf/1702.08608.pdf
• Structured data
– No image, video and sound > deep learning typically not used.
– Tabular data and supervised ML.
• Auto ML
– H2O Driverless AI (DAI) product (not OSS).
• MLI module
– Solution based on MLI module of H2O Driverless AI.
INTRO

7. Confidential7
Terminology, Scope and Context
INTRO
Model
Model
Training
Feature
Engineering
Data
Integration & Quality
Machine
Learning
Interpretability
End to end

8. Confidential8
Terminology, Scope and Context
INTRO
Model
Model
Training
Feature
Engineering
Data
Integration & Quality
Explainable model
End to end

9. Why explainability matters
Problem statement

10. (Trade-off)
Potential Performance and Interpretability Trade-off
White box model Black box model
Feature engineering + Algorithm(s)
Balance

11. (Trade-off)
Potential Performance and Interpretability Trade-off

12. (Trade-off)
Potential Performance and Interpretability Trade-off

13. (Trade-off)
Age
NumberofPurchases
Lost profits.
Wasted marketing.
“For a one unit increase in age, the number
of purchases increases by 0.8 on average.”
Age
“Slope begins to
decrease here. Act to
optimize savings.”
“Slope begins to
increase here sharply.
Act to optimize profits.”
NumberofPurchase
Exact explanations for
approximate models.
Approximate explanations for
exact models.
Linear models
Machine learning models
Potential Performance and Interpretability Trade-off
Sometimes…

14. Trade-off
Multiplicity
• For a given well-understood dataset there
is usually one best linear model, but…
Multiplicity of Good Models

15. Trade-off
Multiplicity
• … for a given well-understood dataset there are usually
many good ML models. Which one to choose?
• Same objective metrics values, performance, …
• This is often referred to as “the multiplicity of good
models.” -- Leo Breiman
Multiplicity of Good Models

16. Trade-off
Multiplicity
Fairness
• Gender
• Age
• Ethnicity
• Health
• Sexual behavior
• Avoid discriminatory models and
remediate disparate impact.
Fairness and Social Aspects

17. Trade-off
Multiplicity
Fairness
Trust
• Dataset
vs.
real world
• ML adoption
• Introspection
• Sensitivity
• OOR
• Diagnostics
• “Debugging”
Trust of model producers & consumers
Source: http://www.vias.org/tmdatanaleng/

18. Trade-off
Multiplicity
Fairness
Trust
Security
• Goal: compromise model integrity
• Attack types:
– Exploratory
– Surrogate model trained to identify vulnerabilities ~ MLI.
– Trial and error (for specific class) x indiscriminate attacks.
– Causative
– Models trained w/ adversary datasets.
– Local model > adversarial instance > target model.
– Standard / continuous learning.
– Integrity (compromise system integrity)
– False negative instance e.g. fraud passes check.
– Availability (compromise system availability)
– False positive instance e.g. blocks access to legitimate instances.
Security and Hacking

19. Trade-off
Multiplicity
Fairness
Trust
Security
Regulation
• Legal requirements
– Banking, insurance, healthcare, …
• Predictions explanation
– Decisions justification (reason codes*, …).
• Fairness
• Security
• Accuracy first vs. interpretability first
– Competitions vs. real world.
Regulated & Controlled Environments

20. Trade-off
Multiplicity
Fairness
Trust
Security
Regulation
• Balance Performance and interpretability.
• Multiplicity of good models.
• Fairness and social aspects.
• Trust of model producers and consumers.
• Security and hacking.
• Regulated/controlled environments .
Explainability Matters

21. What’s needed
Building human-centered, low-risk models

22. Confidential23 Confidential23
• Big picture.
• Interpretability focused.
• MLI module demo only.
• DAI auto ML models.*
• MLI UCs coverage by DAI.
• Techniques and algorithms.
• Possible workflow.
• IID and TS.
• Where MLI module fits in E2E.
Building Human-Centered, Low-Risk Models
*) MLI module is not limited to DAI’s models

23. Confidential24
Building Human-Centered, Low-Risk Models
Load data

24. Confidential25
Explanatory Data Analysis and Visualization
EDALoad data

25. Confidential26
Explanatory Data Analysis and Visualization
EDALoad data

26. Confidential27
Explanatory Data Analysis and Visualization
EDALoad data

27. Confidential28
Explanatory Data Analysis and Visualization
EDALoad data

28. Confidential30
Feature Engineering (Manual & Auto ML)
EDA
Feature engineering
Load data
Black box
model

29. Confidential31
Feature Engineering
EDA
Feature engineering
Load data
White box
model

30. Confidential32
Model Choice: Constrained, Simple, Fair
EDA
Feature engineering
Models
Load data
Black box
model
White box
model
Holy Grail

31. Confidential34
Model Choice: Constrained, Simple, Fair
EDA
Feature engineering
Models
Load data
Black box
model
White box
model
GLM (log regr.), Monotonic GBM (DT), XNN, …

32. Confidential35
Model Choice
EDA
Feature engineering
Models
Load data
Interpretability
Ensemble
Level
Target
Transformation
Feature Engineering
Feature
Pre-Pruning
Monotonicity
Constraints
1 - 3 <= 3 None Disabled
4 <= 3 Inverse None Disabled
5 <= 3 Anscombe
Clustering (ID, distance)
Truncated SVD
None Disabled
6 <= 2
Logit
Sigmoid
Feature selection Disabled
7 <= 2 Frequency Encoding Feature selection Enabled
8 <= 1 4th
Root Feature selection Enabled
9 <= 1
Square
Square Root
Bulk Interactions (add,
subtract, multiply,
divide)
Weight of Evidence
Feature selection Enabled
10 0
Identity
Unit Box
Log
Date Decompositions
Number Encoding
Target Encoding
Text (TF-IDF,
Frequency)
Feature selection Enabled

33. Confidential36
Model Choice
EDA
Feature engineering
Models
Load data

34. Confidential37
Traditional Model Assessment and Diagnostics
EDA
Feature engineering
Models
Assessment
Load data

35. Confidential38
Traditional Model Assessment and Diagnostics
EDA
Feature engineering
Models
Assessment
Load data

36. Confidential39
Traditional Model Assessment and Diagnostics
EDA
Feature engineering
Models
Assessment
Load data Experiment summary (document + YAML) + AutoDoc

37. Confidential40
Traditional Model Assessment and Diagnostics
EDA
Feature engineering
Models
Assessment
Load data

38. Confidential41 Confidential41
• Post-hoc model debugging
– What-if, sensitivity analysis (accuracy).
• Post-hoc explanations
– Reason codes.
• Post-hoc bias assessment and remediation
– Disparate impact analysis.
Post-hoc Model Explanations and Debugging
EDA
Feature engineering
Models
Assessment
Explanations
Post-hoc
Bias remediation
Post-hoc
Model debugging
Post-hoc
Load data

39. Confidential42
Human Review
EDA
Feature engineering
Models
Assessment
Human review
Explanations
Post-hoc
Bias remediation
Post-hoc
Model debugging
Post-hoc
Load data
Semantics

40. Confidential43
Human Review
EDA
Feature engineering
Models
Assessment
Human review
Explanations
Post-hoc
Bias remediation
Post-hoc
Model debugging
Post-hoc
Load data
Semantics

41. Confidential44
Iterative Improvement
EDA
Feature engineering
Models
Assessment
Human review
Deployment
Iteratetoimprovemodel
Explanations
Post-hoc
Bias remediation
Post-hoc
Model debugging
Post-hoc
Load data
Semantics

42. How
Explaining models - MLI module deep dive

43. Confidential46
H2O Driverless AI’s MLI module

44. Confidential47
IID and Time Series

45. Confidential48
H2O Driverless AI’s MLI module
Global approximate model behavior/interactions
Global feature importance
Shapley
DT
Global feature behavior
Reason codes
K-LIME
PDP
Model
Local feature importance
Local feature behavior
ICE
Local approximate model behavior
Model
RF

46. Confidential49
Demo Dataset: Credit Card (IID)
Column Name Description
ID ID of each client
LIMIT_BAL Amount of given credit in NT dollars (includes
individual and family/supplementary credit)
SEX Gender (1=male, 2=female)
EDUCATION (1=graduate school, 2=university, 3=high school,
4=others, 5=unknown, 6=unknown)
MARRIAGE Marital status (1=married, 2=single, 3=others)
AGE Age in years
PAY_x {1, …,6} Repayment status in August, 2005 – April, 2005 (-1=paid
duly,1=payment delay for 1 month, …,8=payment delay for
8 months)
BILL_AMTx {1, …, 6} Amount of bill statement in
September, 2005 – April, 2005 (NT dollar)
PAY_AMTx {1, …, 6} Amount of previous payment in September,
2005 – April, 2005 (NT dollar)
default_payment_
next_month
Default payment (1=yes, 0=no)
Target
Education,
Marriage, Age,
Sex,
Repayment
Status, Limit
Balance, ...
Features
Default
Payment Next
Month
(Binary)
Predictions
Probability
(0...1)

47. Confidential50
• Challenge:
– Black-box models
– Original vs. transformed features.
• Solution: Surrogate models
– Pros
– Increases any black-box model’s
interpretability
– Time complexity
– Cons
– Accuracy
Global Approximate Model Behavior/Interaction

48. Confidential51
Surrogate Models
Model
Surrogate Models
Model
training
Model
training

49. Confidential52
• Challenges:
– Black-box models
– Original vs. transformed features
• Solutions:
– Surrogate model: RF (introspection)
– Pros:
– Original features
– Time complexity
– Cons:
– Accuracy
Global Feature Importance: Random Forest

50. Confidential53
• Challenges:
– Black-box models
– Original vs. transformed features
• Solutions:
– Original (DAI) Model Introspection
– Pros:
– Accuracy
– Cons:
– Transformed features
– Global only
Global Feature Importance: Original Model

51. Confidential54
• Challenge
– Black-box models
– Original vs. transformed features
• Solutions:
– Shapley values
– Pros:
– Accuracy
– Math correctness
– Cons:
– Time complexity
– Transformed features
Global Feature Importance: Shapley Values

52. Confidential55
• Lloyd Shapley
– Americal mathematician who won Nobel prize in
2012 (Economics).
– Shapley values was his Ph.D. thesis written
in 50s.
• Shapley values:
– Supported by solid mathematical (game) theory.
– Calculation has exponential time complexity
(number of coalitions) .
– Typically unrealistic to compute in real world.
– Can be computed in global or local scope.
– Guarantee fair distribution among features in
the instance.
– Does not work well in sparse cases, all features
must be used.
– Return single value per feature, not a model.
Shapley Values

53. Confidential56
Feature importance: Leave One Covariate Out
• UC:
– Complete other feature
importance charts with bias
tendency
• Challenge:
– Black-box models
• Solution:
– LOCO

54. Confidential57
Feature importance: Leave One Covariate Out

55. Confidential58
• Methods
– Surrogate models:
– RF (introspection)
– Leave One Covariate Out (LOCO)
– Original model (introspection)
– Shapley values
Global Feature Importance

56. Confidential59
Global Feature Behavior: Partial Dependence Plot
• Solution: Surrogate model PDP
– Pros
– Time complexity
– Original features
– White/black model interpretability
– Cons
– Accuracy

57. Confidential60
• Solution: Surrogate model PDP
– Pros
– Time complexity
– Original features
– White/black model interpretability
– Cons
– Accuracy
Global Feature Behavior: Partial Dependence Plot
Model
Prediction
Xj

58. Confidential61
PDP: Character of the Feature Behavior

59. Confidential62
Reason codes: Local Feature Importance
Global approximate model behavior/interactions
Global feature importance
Shapley
DT
Global feature behavior
Reason codes
K-LIME
PDP
Model
Model
RF

60. Confidential63
Reason codes: Local Feature Importance
• UCs:
– Predictions explanations
– Legal
– Debugging
– Drill-down,
– …
• From global to local scope
• Surrogate methods:
– K-LIME (K-means)
– LIME-SUP (trees)

61. Confidential64
LIME: Local Interpretable Model-agnostic Explanations
Source: https://github.com/marcotcr/lime
Weighted
explanatory
samples
• Weighted linear surrogate model used to explain
non-linear decision boundary in local region.
• Single prediction.
• example:
– Set of explainable records are scored using
the original model.
– To interpret a decision about another
record, the explanatory records are
weighted by their closeness to that record.
– L1 regularized linear model is trained on this
weighted explanatory set.
– The parameters of the linear model then
help explain the prediction for the selected
record.

62. Confidential65
K-LIME: Clustered LIME

63. Confidential66
Reason codes: Local Feature Importance
• UCs:
– Predictions explanations
– Legal
– Debugging
– Drill-down,
– …
• From global to local scope
• From global explanatory model
to cluster-scoped explanatory
model.
• UCs:
– Predictions explanations
– Legal
– Debugging
– Drill-down,
– …
• From global to local scope

64. Confidential67
Reason codes: Local Feature Importance
• Challenges:
– Black-box models
– Original vs. transformed features
• Solutions:
– Surrogate model: K-LIME
– Pros:
– Original features
– Time complexity
– Cons:
– Accuracy
reason
code

65. Confidential68
• UC:
– Particular instance explanation
– Note path segments thickness.
• Challenge:
– Black-box models
• Solution: Surrogate models
– Pros
– Black-box models interpretability
– Time complexity
– Cons
Local Approximate Model Behavior/Interaction

66. Confidential69
• Mean absolute value vs.
local contributions
• Challenge
– Black-box models
– Original vs. transformed features
• Solutions:
– Surrogate models:
– RF (introspection)
– Leave One Covariate Out (LOCO)
– Shapley values
Local Feature Importance

67. Confidential70
Local Feature Behavior: ICE
• Solution: Surrogate model ICE
– Pros
– Time complexity
– Original features
– White/black model interpretability
– Cons
– Accuracy
(dotted line vs. gray dot discrepancy)

68. Confidential71
ICE: Individual Conditional Expectations
• Solution: Surrogate model ICE
– Pros
– Time complexity
– Original features
– White/black model interpretability
– Cons
– Accuracy
Model
Prediction
Xj

69. Confidential72
AutoDoc

70. Confidential73
• Time series experiments
– Test dataset
• Explainability:
– Original model
– Global and per-group
– Forecast horizon
– Feature importance
– Per-group
– Local Shapley values
MLI for Time Series

71. Confidential74
MLI Cheatsheet
https://github.com/h2oai/mli-resources/blob/master/cheatsheet.png

72. Confidential75
MLI Functional Architecture
Local k-LIME
<<diagram>>
T: RMSE, R2
E: how much are
local GLMs preds off
user model MU
(Ŷ
curve vs. ŶS
dots)
T: math behind Sh.
Local Shapleys
<<report>>
E: how much fU
i
influences preds in
case of ŷi (+/- contr.
coefs. via MU
GBM)
T: math behind Sh.
Local Shapleys
<<report>>
E: how much fU
i
influences preds in
case of ŷi (+/- contr.
coefs. via MU
GBM)
ICE (local char.)
<<diagram>>
T: N/A
E: direct/inverse/no
proportion (correl.)
of fi in case of
particular ŷi
ICE (local char.)
<<diagram>>
T: N/A
E: direct/inverse/no
proportion (correl.)
of fi in case of
particular ŷi
Local k-LIME
<<diagram>>
T: RMSE, R2
E: how much are
local GLMs preds off
user model MU
(Ŷ
curve vs. ŶS
dots)
Local reason codes
<<report>>
T: RMSE, R2
E: how much fi
influences preds in
case of ŷi (+/- contr.
coefs. in local GLM)
Local reason codes
<<report>>
T: RMSE, R2
E: how much fi
influences preds in
case of ŷi (+/- contr.
coefs. in local GLM)
ŶS
RF fiŶS
RF fi
XfiXfi
ŶS
k-lime
ŶS
k-lime
MS
local k-LIME
predict(X)
MS
local k-LIME
predict(X)
XFi
ŶU
FiŶU
Fi
MS
local k-LIME
1 x GLM
<<model>>
MS
local k-LIME
1 x GLM
<<model>>
XFi
DAI: create user model fit(X,Y)
X Y
XU
ŶU
H2O-3: create surrogate models: DT.fit(X, ŶU
), 1+k * GLM.fit(X, ŶU
) and RF.fit(X, ŶU
)
X
Figure: MLI-2 functional architecture flow diagram
Xfi
Mu
predict(Xfi) ~ specific
values (bins) fixed for fi
ŶU
fi
MU
<<model>>
Transformed
features
MU
: use user model to predict(X)
MS
global GLM
predict(X)
X
F
FU
MU
GBM
<<model>>
Local surrogate DT
<<diagram>>
T: RMSE, R2
E: how fis influence
predictions in case of
particular ŷi +
interactions (typical
path in DT)
Global surrogate DT
<<diagram>>
T: RMSE, R2
E: how fis influence
predictions of Ŷ +
interactions
(typical path in DT)
Global reason codes
<<report>>
T: RMSE, R2
E: how much fi
influences preds in
case of ŷi (+/- contr.
coefs. in glob. GLM)
Local LOCO
<<report>>
T: bias (plot, contrib.)
Global LOCO
<<report>>
ŶS
global GLM
MS
RF
1 x RF
<<model>>
MS
global GLM
1 x GLM
<<model>>
MS
local GLM
k x GLM
<<model>>
PDP (global char.)
<<diagram>>
T: N/A
MS
local GLM
predict(X)
X
ŶS
local GLM
Global feature importance
<<diagram>>
T: N/A
E: how much fi influences
predictions of Ŷ
(importance not
contribution, depth in RF)
MS
RF
predict(X)
X
ŶS
RF
MS
RF
predict(Xfi)
Xfi
ŶS
RF fi
E: how much fi
contributes to
predictions in case of
ŷi (leave fi in vs out,
def p RF)
T: N/A
E: how much fi
contributes to
predictions of Ŷ (leave
fi in vs. out difference
for all ŷi)
Local reason codes
<<report>>
T: RMSE, R2
E: how much fi
influences preds in
case of ŷi (+/- contr.
coefs. in local GLM)
Global k-LIME
<<diagram>>
T: RMSE, R2
E: how much are global
GLM preds off for user
model MU
(Ŷ curve vs.
ŶS
dots) + MU
linearity
quantification
Local k-LIME
<<diagram>>
T: RMSE, R2
E: how much are
local GLMs preds off
user model MU
(Ŷ
curve vs. ŶS
dots)
ICE (local char.)
<<diagram>>
T: N/A
E: direct/inverse/no
proportion (correl.)
of fi in case of
particular ŷi preds
E: direct/inverse/no
proportion (correl.)
of fi across all Ŷ
preds
T: bias (plot, contrib.), math
Local Shapley
<<report>>
E: how much transf.
fU
i influences preds
in case of ŷi (contr.
coefs. via MU
GBM)
T: math behind Sh.
Global Shapley
<<report>>
E: how much transf.
fU
i influences preds
in Ŷ (+/- contr.
coefs. via MU
GBM)
Local feature importance
<<diagram>>
T: N/A
E: how much fi influences
predictions in case of
particular ŷi (importance not
contribution, unsigned LOCO
based)
Global feature importance
<<diagram>>
T: N/A
E: how much transf. fU
i
influences predictions of Ŷ
(importance not contrib.,
depth in GBM trees)
MU
GBM
<<model>>
MS
DT
1 x DT
<<model>>
MS
DT
predict(X)
X
ŶS
DT
NewMLI-2PDP/ICEcalculation
Old

73. Conclusion
Takeaways

74. TAKEAWAYS
• ML interpretability matters.
• Multiplicity of good models.
• H2O Driverless AI has interpretability.
• Control model interpretability end to end.
• Prefer interpretable models.
• Test both your model and explanatory SW.
• Use synergy of local & global techniques.
• Shapley values.

75. MLI TEAM
Patrick Navdeep Mateusz
Zac Laco Martin

76. Thank you!

77. ConfidentialConfidential
Resources
Books, articles, links and Git repos,

78. Confidential81 Confidential81
• https://www.h2oai.com/explainable-ai/
• Booklets:
– Machine Learning Interpretability with DAI
– Ideas on Interpreting Machine Learning
• Driverless AI’s MLI module cheatsheet
• MLI presentations:
– MLI walkthrough by Patrick Hall
– Human Friendly Machine Learning by Patrick Hall
• GitHub repositories:
– MLI Resources
– H2O Meetups
Resources