Validating an ML model with train-test accuracy metrics offers an initial understanding of viability but generating consistent inferencing with contextual business goals requires understanding how the deployed model works in different nature and how they will behave in case of soft data drift. In this talk, I will try to go through different explainability methods and how to employ them and how the choice of type of models affects or affects the interpretability in production inferencing.

1. 20th May, 2023 Bengaluru

2. 20th May, 2023 Bengaluru
Speaker
Saradindu Sengupta
Senior ML Engineer @Nunam
Where I work on building learning systems to
forecast health and failure of Li-ion batteries.
Interpretable ML in production

3. Table of Contents
1. Introduction
2. When and Why model understanding
3. What is model interpretability
a. Journey of a model
b. Interpretability framework
4. Achieving model understanding
a. Inherently interpretable models
b. Post-hoc explanations
5. What-if Toolkit

4. Introduction - Kick-off
Predict Wolf vs Husky [6]
Only 1 mistake!

5. Introduction - kick-off
Predict Wolf vs Husky - A great snow detector [6]
A great snow detector…

6. When and Why Model Understanding?
Not all applications require model understanding
1. E.g., ad/product/friend recommendations
2. No human intervention
Model understanding not needed because:
1. Little to no consequences for incorrect predictions
2. Problem is well studied and models are extensively validated in real-world applications

7. When and Why Model Understanding?
High-stakes decision-making settings
1. Impact on human lives/health/finances
2. Settings relatively less well studied, models not extensively validated
Accuracy alone is no longer enough
1. Train/test data may not be representative of data encountered in practice
Auxiliary criteria are also critical:
1. Nondiscrimination
2. Right to explanation
3. Safety

8. What is model interpretability
Journey of a model
X1
X2
X3
.
.
xn
y,,
y*
Evaluation
Metrics
Users
But can you trust your model?
Will it work in deployment?

9. What is model interpretability
Interpretability Framework
1. Trust
a. A prerequisite for humans to trust models
2. Causality
a. Learned associations between variables and outcomes
3. Transferability
a. How models might fare when the test environment shifts from the training environment.
4. Fair and Ethical Decision Making
a. Algorithmic decisions must be explainable, contestable, and modifiable
Geared towards supervised learning [4]. For understanding interpretability in reinforcement learning [5], which studies the
human interpretability of robot actions.

10. Achieving Model Understanding
1. Build inherently interpretable predictive models

11. Achieving Model Understanding
2. Explain pre-built models in a post-hoc manner
Explainer

12. Achieving Model Understanding
Inherently Interpretable Models vs. Post hoc Explanations [9]
Example
In certain scenarios, accuracy-interpretability trade will may exist

13. Achieving Model Understanding
Inherently Interpretable Models vs. Post hoc Explanations [9]
complex models might achieve higher accuracy
can build interpretable + accurate models

14. Achieving Model Understanding
Inherently Interpretable Models
1. Rule Based Models
a. Bayesian Rule List
b. Decision sets
2. Risk Scores
a. Widely used in medicine, criminal justice system,
regulated industry such as insurance and loan
3. Linear models
a. Linear regressions
4. Generalized Additive Models
5. Prototype Based Models
6. Tree based models
a. Decision Tree
b. Tree-based ensemble Model

15. Achieving Model Understanding
Post hoc Explanations
LIME (Local Interpretable Model-Agnostic Explanations) [6]
Given an example, x, LIME attempts to fit an interpretable model locally that is faithful to the output of the original model,
f(x), in a neighborhood around x
1. Sample points around xi
2. Use model to predict labels for each sample
3. Weigh samples according to distance to xi
4. Learn simple linear model on weighted samples
5. Use simple linear model to explain

16. Achieving Model Understanding
Post hoc Explanations
SHAP (SHapley Additive exPlanations) [10]
It tries to add 3 attributes that we want in an interpretable model
1. Local accuracy
2. Missingness
3. Consistency
xi
P(y) = 0.9
xi
P(y) = 0.8
M(xi
, O) = 0.1
O
O/xi

17. What-if Toolkit [11]
Google Colab || Data Set: UCI Census Income Dataset || GitHub Repository with Tutorials
Vertex AI integration

18. References
1. Zachary C. Lipton, “The Mythos of Model Interpretability”, 2016, [https://arxiv.org/abs/1606.03490]
2. Finale Doshi-Velez, Been Kim, “Towards A Rigorous Science of Interpretable Machine Learning”, 2017,
[https://arxiv.org/abs/1702.08608]
3. S. Ghanta et al., "Interpretability and Reproducability in Production Machine Learning Applications," 2018 17th IEEE
International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, USA, 2018, pp. 658-664, doi:
10.1109/ICMLA.2018.00105.
4. Lou,Yin, Caruana, Rich, and Gehrke, Johannes. “Intelligible models for classification and regression”.In KDD, 2012.
5. Dragan, AncaD, Lee, Kenton CT, and Srinivasa, Siddhartha S., “Legibility and predictability of robot motion. In Human-Robot
Interaction(HRI), 2013 8th ACM/IEEE International Conference on. IEEE, 2013.
6. Ribeiro et. al. 2016[https://arxiv.org/abs/1602.04938]
7. Ribeiro et al. 2018[https://homes.cs.washington.edu/~marcotcr/aaai18.pdf]
8. Lakkaraju et. al. 2019 [https://dl.acm.org/doi/10.1145/3306618.3314229]
9. Machine Learning Explainability Workshop, Stanford University
[https://youtube.com/playlist?list=PLoROMvodv4rPh6wa6PGcHH6vMG9sEIPxL]
10. Lundberg, S., & Lee, S. (2017). A Unified Approach to Interpreting Model Predictions. ArXiv. /abs/1705.07874
[https://arxiv.org/abs/1705.07874]
11. J. Wexler, M. Pushkarna, T. Bolukbasi, M. Wattenberg, F. Viégas and J. Wilson, "The What-If Tool: Interactive Probing of
Machine Learning Models," in IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 1, pp. 56-65, Jan. 2020,
doi: 10.1109/TVCG.2019.2934619.[https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8807255]

19. Thank You
linkedin/saradindusengupta
@iamsaradindu