This document discusses interpretable machine learning and explainable AI. It begins with definitions of key terms and an overview of interpretable methods. Deep learning models are often treated as "black boxes" that are difficult to interpret. Interpretability can be achieved by using inherently interpretable models like linear models or decision trees, adding attention mechanisms, or interpreting models before, during or after building them. Later sections discuss specific interpretable techniques like understanding data through examples, MMD-Critic for learning prototypes and criticisms, and visualizing convolutional neural networks to understand predictions. The document emphasizes the importance of interpretability and explains several approaches to make machine learning models more transparent to humans.

1. Introduction to Interpretable
Machine Learning
Presented by Giang Nguyen
KAIST, Nov 2019

2. Terminologies
- Interpretable ML
- Explanable AI
- X – AI
2

3. BIRD VIEW OVER EXPLAINABLE AI

4. Deep Learning as Blackbox
While powerful, deep learning models are difficult to interpret, and thus
often treated as a blackbox.
4

5. Interpretability
Interpretation is the process of giving explanations to humans.
Interpretability is not a well-defined concept
5

6. Types of Interpretable Methods
We can interpret the model either before building the model, when
building it, or after building a model.
Most interpretation methods for DNNs interpret the model after it is built.
6

7. Interpretation When Building a Model
7

8. Using Inherently Interpretable Models
(Sparse) linear models and decision trees are inherently interpretable.
8

9. Attention Mechanisms
Attention mechanisms guide deep neural networks to focus on
relevant input features, which allows to interpret how the model made
certain predictions.
9
[Bahdanau et al. 15] Neural Machine Translation by Jointly Learning to Align and Translate, ICLR 2015

10. Limitation of Conventional Attention Mechanisms
Conventional attention models may allocate attention inaccurately since
they are trained in a weakly-supervised manner.
The problem becomes more prominent when a task has no one-to-one
mapping from inputs to the final predictions.
10

11. Limitation of Conventional Attention Mechanisms
This is because the conventional attention mechanisms do not consider
uncertainties in the model and the input, which often leads to
overconfident attention allocations.
Such unreliability may lead to incorrect predictions and/or interpretations
which can result in fatal consequences for safety-critical applications.
11

12. Uncertainty Aware Attention (UA)
12

13. Uncertainty Aware Attention (UA)
13
Multi-class classification performance on the three health records datasets

14. Info-GAN
14
There are structures in the noise vectors that have meaningful and
consistent effects on the output of the generator.
However, there’s no systematic way to find these structures. The only
thing affecting to the generator output is the noise input, so we have no
idea how to modify the noise to generate expected images.

15. Info-GAN
15
The idea is to provide a latent code, which has meaningful and consistent
effects on the output - disentangled representation
The hope is that if you keep the code the same and randomly change the
noise, you get variations of the same digit.

16. Info-GAN
16
c1 ∼ Cat(K = 10, p = 0.1)

17. Interpretation After Building a Model
17

18. Understanding Black-Box Predictions
Given a high-accuracy blackbox model and a prediction from it, can we
answer why the model made a certain prediction?
[Koh and Liang 17] tackles this question by training a model’s prediction through its learning algorithm
and back to the training data.
To formalize the impact of a training point on a prediction, they ask the counterfactual:
What would happen if we did not have this training point or if its value were slightly changed?
18
[Koh and Liang 17] Understanding Black-box Predictions via Influence Functions, ICML 2017

19. Interpretable Mimic Learning
This framework is mainly based on knowledge distillation from Neural
Networks.
However, they use Gradient Boosting Trees (GBT) instead of another neural
network as the student model since GBT satisfies our requirements for
both learning capacity and interpretability.
19[Che et al. 2016] Z. Che, S. Purushotham, R. Khemani, and Y. Liu. Interpretable Deep Models for
ICU outcome prediction, AMIA 2016.
Knowledge distillation
G. Hinton et al. 15

20. Interpretable Mimic Learning
The resulting simple model works even better than the best deep learning
model – perhaps due to suppression of the overfitting.
20[Che et al. 2016] Z. Che, S. Purushotham, R. Khemani, and Y. Liu. Interpretable Deep Models for
ICU outcome prediction, AMIA 2016.

21. Visualizing Convolutional Neural Networks
Propose Deconvolution Network (deconvnet) to inversely map the feature
activations to pixel space and provide a sensitivity analysis to point out
which regions of an image affect to decision making process the most.
21
[Zeiler and Fergus 14] Visualizing and Understanding Convolutional Networks, ECCV 2014

22. Prediction difference analysis
22
The visualization method shows which pixels of a specific input image are
evidence for or against a prediction
[Zintgraf et al. 2017] Visualizing Deep Neural Network Decisions: Prediction Difference Analysis, ICLR 2017
Shown is the evidence for (red) and against (blue) the prediction.
We see that the facial features of the cockatoo are most supportive for the decision, and
parts of the body seem to constitute evidence against it.

23. Interpretation Before Building a Model
23

24. Understanding Data Through Examples
[Kim et al. 16] propose to interpret the given data by providing examples
that can show the full picture – majorities + minorities
[Kim et al. 16] Examples are not Enough, Learn to Criticize! Criticism for Interpretability 24

25. INTRODUCTION

26. AI is data-driven, what we get is what we have
26

27. Understanding data through examples
27

28. Understanding data through examples
28

29. Understanding data through examples
29

30. Understanding data through examples
30

31. Understanding data through examples
31

32. Understanding data through examples
33

33. Understanding data through examples
34

34. Ideas of paper
35

35. Related Work

36. 37
Over-generalization
Over-generalization is consistent with evolutionary
theory [Zebrowitz ‘10, Schaller’ 06]
algorithms can help against over-generalization

37. 38
Venn diagram of related works

38. MMD-critic

39. 40
Approach

40. 41
Maximum Mean Discrepancy (MMD)

41. 42
MMD-Critic: Learning Prototypes and Criticisms

42. Experiments

43. 44
Results

44. 45
Prototype-based classification
• Use the learned prototypes for classification (nearest-neighbor)

45. 46
Example Prototypes and Criticisms
• USPS Digits Dataset
Unrecognizable

46. 47
Example Prototypes and Criticisms
• ImageNet Dataset – 2 breeds of dog

47. 48
Pilot study with human subjects
Definition of interpretability: A method is interpretable if a user can
correctly and efficiently predict the method’s results.
Task: Assign a new data point to one of the groups using 1) all images
2) prototypes 3) prototypes and criticisms 4) small set of randomly
selected images

48. 49
Pilot study with human subjects

49. Conclusion

50. Take-home messages
51
• There are three types of Interpretable Methods, but mostly after building
models
• Criticism and prototypes are equally important and are a step towards
improving interpretability of complex data distributions
• MMD-critic learns prototypes + criticisms that highlight aspects of
data that are overlooked by prototypes.

51. Discussion
52
• If we have the insight into a dataset, can we really build a better model?
Human intuition is biased and not realiable!

52. Gap in Interpretable ML research
53
• Limited work to explain the operation of RNNs, only CNN. Attention
mechanism is not enough. Especially in multimodal network (CNN +
RNN), this kind of research is more necessary

53. Thank you for your attention!