ODSC APAC 2022 : Explainable Machine Learning - A Human Centric Perspective By Aditya Bhattacharya - 7 September 2022 ODSC APAC

1. Explainable AI Researcher
KU Leuven
Explainable Machine Learning
A Human-Centric Perspective
September 7th, 2022

2. About Me
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
2
Aditya
Bhattacharya
I am currently working as an Explainable AI Researcher at KU Leuven,
Belgium with an overall experience of 7 years in Data Science, Machine
Learning, IoT & Software Engineering. Prior to his current role, Aditya has
worked in various roles in organizations like West Pharma, Microsoft & Intel
to democratize AI adoption for industrial solutions. As the AI Lead at West
Pharma, he had contributed to forming the AI Centre of Excellence,
managing & leading a global team of 10+ members focused on building AI
products.
Apart from my day job, I am an AI Researcher, Executive Member, Faculty
at an NGO called MUST Research https://must.co.in/. I am also a content
creator, adjunct faculty at UpGrad https://www.upgrad.com/ .
Website : https://aditya-bhattacharya.net/
LinkedIn: https://www.linkedin.com/in/aditya-bhattacharya-b59155b6/

3. Key Topics
3
1. Conceptual understanding of XAI methods
• Understanding the need for XAI
• Dimensions of explainability
• Approaches for explainability
• Different types of explainability methods
2. Discussions on existing Python frameworks for explainability
3. ENDURANCE - End User Centric Artificial Intelligence
• Understanding open challenges of XAI
• Industry best practices,
• Bridging the XAI-end user gaps

4. Conceptual understanding of XAI
methods

5. 5
Applied Machine Learning Explainability Techniques, A. Bhattacharya

6. 6
Applied Machine Learning Explainability Techniques, A. Bhattacharya

7. 7
Applied Machine Learning Explainability Techniques, A. Bhattacharya

8. 8
• Data: Data centric explanation methods revolves around the
underlying data that is being modeled. Understanding the data,
identifying its limitations and using conventional data analytics to
generate actionable insights.
• Model: Model-based interpretability techniques often help us to
understand how the input data is mapped to the output predictions
using certain approximation methods.
• Outcomes: The outcome of explainability is about understanding
why and how a certain prediction or decision is made by an ML
model and how does the change in input changes the output of the
model.
• End users: The final dimension of explainability is all about
creating the right level of abstraction and including the right
amount of details for the final consumers of the ML models so that
the outcomes are reliable and trustworthy for any non-technical
end-user
Applied Machine Learning Explainability Techniques, A. Bhattacharya

9. 9

10. 1 0
Problem, Data,
Audience
Post Hoc
Analysis
Model
Predictive
Accuracy
Descriptive
Accuracy
Iterative
Explainability
Pre Hoc
Analysis

11. 1 1

12. 1 2
Local vs Global
Intrinsic vs
Extrinsic
Model-specific vs
Model-agnostic
Model-Centric vs
Data-Centric
Different Explanation Methods

13. Model
Explanations
Methods
Results
Visualizations
Influence
Methods
Example
Based
Methods
Knowledge
Extractions
1 3
Using Surrogate models like linear
models or decision trees to explain
complex models
Estimates the impact or
relevance of features.
Extracting information
from input and the output
Select instances of the datasets that
explains the behaviour of the model

14. 1 4
Applied Machine Learning Explainability Techniques, A. Bhattacharya

15. Popular Python frameworks for XAI

16. Popular frameworks for XAI
1 6
LIME
• Local
Interpretable
Model-agnostic
Explanations is
interpretability
framework that
works on structured
data, text and
image classifiers.
SHAP
• SHAP (SHapley
Additive
exPlanations) is a
game theoretic
approach to explain
the output of any
machine learning
model.
DALEX
• moDel Agnostic
Language for
Exploration
(DALEX) xrays any
model and helps to
explore and explain
its behaviour.
Explainer
dashboards
• Explainerdashboa
rd makes it
convenient to
quickly deploy a
dashboard web app
that explains ML
model.
TCAV
• Testing with
Concept Activation
Vectors (TCAV) is a
new interpretability
method to
understand what
signals your neural
networks models
uses for prediction.

17. 1 7
• Behind the workings of LIME lies the assumption that every complex model is linear on a local scale.
LIME tries to fit a simple model around a single observation that will mimic how the global model
behaves at that locality.
• Create the perturbed data and predict the output on the perturbed data
• Create discretized features and find the Euclidean distance of perturbed data to the original
observation
• Convert distance to similarity score and select the top n features for the model
• Create a linear model and explain the prediction

18. 1 8
The lime package is on PyPI.
`pip install lime`

19. 1 9
Applied Machine Learning Explainability Techniques, A. Bhattacharya

20. 2 0
There is a high-speed exact algorithm for tree ensemble methods (Tree SHAP arXiv paper). Fast C++
implementations are supported for XGBoost, LightGBM, CatBoost, and scikit-learn tree models!
• SHAP assigns each feature an importance
value for a particular prediction.
• Its novel components include: the identification
of a new class of additive feature importance
measures, and theoretical results showing
there is a unique solution in this class with a
set of desirable properties.
• Typically, SHAP values try to explain the output
of a model (function) as a sum of the effects of
each feature being introduced into a
conditional expectation. Importantly, for non-
linear functions the order in which features are
introduced matters.
SHAP can be installed from PyPI

21. 2 1
Applied Machine Learning Explainability Techniques, A. Bhattacharya

22. 2 2
The following figure from the KDD 18 paper, Consistent Individualized
Feature Attribution for Tree Ensembles summarizes this in a nice way!
SHAP Summary Plot
SHAP Dependence Plots
SHAP Gradient Explainer for Images
Applied Machine Learning Explainability Techniques, A. Bhattacharya

23. Project Source –
https://github.com/ModelOriented/DALEX

24. 2 4
Source Project -
https://github.com/oegedijk/e
xplainerdashboard

25. 2 5
Testing with Concept Activation Vectors (TCAV) is
a new interpretability method to understand what
signals your neural networks models uses for
prediction.
What's special about TCAV
compared to other methods?
TCAV instead shows importance of high
level concepts (e.g., color, gender, race)
for a prediction class - this is how humans
communicate!
TCAV gives an explanation that is generally true for a class of interest, beyond one image (global
explanation).
For example, for a given class, we can show how much race or gender was important for classifications in
InceptionV3. Even though neither race nor gender labels were part of the training input!
pip install tcav https://github.com/tensorflow/tcav

26. 2 6
The Concept Activation Vectors (CAVs) provide an interpretation of a neural net’s internal state
in terms of human-friendly concepts. TCAV uses directional derivatives to quantify the degree to
which a user-defined idea is vital to a classification result–for example, how sensitive a prediction
of “zebra” is to the presence of stripes.
TCAV essentially learns ‘concepts’
from examples. For instance, TCAV
needs a couple of examples of ‘female’,
and something ‘not female’ to learn a
“gender” concept. The goal of TCAV is
to determine how much a concept (e.g.,
gender, race) was necessary for a
prediction in a trained model even if the
concept was not part of the training.
Applied Machine Learning Explainability Techniques, A. Bhattacharya

27. Research question to consider …

28. All these frameworks are great
and can bring explainability to a
great extent, but can non-expert
consumers of AI models
interpret these explanation
methods?
2 8

29. ENDURANCE - End User Centric Artificial Intelligence

30. 3 0
Applied Machine Learning Explainability Techniques, A. Bhattacharya
• Shifting focus between the model developer and the end user
• Lack of stakeholder participation
• Application-specific challenges
• Lack of quantitative evaluation metrics
• Lack of actionable explanations
• Lack of contextual explanations

31. 3 1
Applied Machine Learning Explainability Techniques, A. Bhattacharya

32. 3 2
Applied Machine Learning Explainability Techniques, A. Bhattacharya
• Identify the target audience of XAI and their usability context
• Shortlisting the XAI techniques based on the user's needs
• Human-centered XAI: An iterative process of translating and
evaluating XAI in specific domains involving the end user
• The importance of the feedback loop in XAI
• The importance of scalability in the design process
• Toggling between the data, the interface, and actionable insights

33. 3 3
Applied Machine Learning Explainability Techniques, A. Bhattacharya
Adopting a data-first approach for explainability

34. 3 4
Applied Machine Learning Explainability Techniques, A. Bhattacharya
Emphasizing prescriptive insights for explainability

35. 3 5
Applied Machine Learning Explainability Techniques, A. Bhattacharya
Emphasizing interactive machine learning for explainability

36. Summary
1. Conceptual understanding of XAI methods
2. Discussions on existing Python frameworks for
explainability
3. ENDURANCE - End User Centric Artificial Intelligence
3 6
https://github.com/PacktPublishing/Applied-Machine-
Learning-Explainability-Techniques
Make ML models explainable and trustworthy for practical
applications using LIME, SHAP and more
Amazon link - https://amzn.to/3OAZZPf

37. Thank you
Aditya Bhattacharya
https://aditya-bhattacharya.net/
aditya.bhattacharya2016@gmail.com
https://www.linkedin.com/in/aditya-bhattacharya-b59155b6/
3 7