This work is highly influenced by work previously completed by Zachary Lipton in Mythos of Model Interpretability. Essentially we are arguing that as long as there's no consensus and formal standardisation of what people mean by interpretability it will prevent us from having a pragmatic and influential progress in this direction. At the time of the presentation there was no consesus on validation metrics, datasets or methodologies to evaluate and compare interpretability methods in the literature. We highly emphasised the need of an axiomatic and formal approach relating to earlier efforts in interpretability in fuzzy systems in order to enforce the healthy habit of thinking about formal definitions and standardisations.

1. Wednesday, 27th March
A Post-Hoc Categorisation
of Predictive Models
John Mitros
University College Dublin
ioannis.mitros@insight-centre.org

2. Outline
• Introduc)on
• Overview of interpretability/explainability
• Post-hoc approaches for interpretability
• Common themes, connec)ng ideas, general picture
• Not an exhaus)ve survey of all the literature body
• Open challenges and possible future direc)ons
• Examples and use cases
• Recent approaches
2

3. Explainable AI
3
Image from https://xaitutorial2019.github.io/

4. Explainable vs. Interpretable
• Explainable ML:
• Post-hoc analysis of black box models
• Interpretable ML:
• Intrinsically interpretable a.k.a transparent
4Rudin, C. & Ertekin, Ş. Math. Prog. Comp. (2018) 10: 659. hNps://doi.org/10.1007/s12532-018-0143-8

5. Interpretability
• It is inherently a mul/faceted no/on whose meaning changes according to
the different applicability scenarios
• Interpretability needs to answer what the model has learned and why it
came to that conclusion
• Defini/on of interpretability:
• “interpretability is the degree to which a human can understand the cause of a
decision” (Miller 2017)
5

6. Interpretability
• Defini&on of interpretability:
• “interpretability is the degree to which a machine can explain the cause of a decision
into coherent logical arguments”
• inherently it involves a bijec&ve process from input to output and vice versa, where
the intermediate steps are transparent to the end user
!" " # → % &ℎ() " % → #
• logical fallacies should be avoided
6

7. Scope of Interpretability
7
Lipton, Z. C. 2016. The Mythos of Model Interpretability. ICML Workshop on Human Interpretability in Machine Learning
(WHI 2016), New York, NY

8. Scope of Interpretability
8

9. Hierarchical Structure of Interpretability
9

10. Examples of Post-hoc Explana2ons
10
Chen, C.; Li, O.; Barne1, A.; Su, J.; and Rudin, C. 2018. This looks like that: Deep learning for interpretable image
recogniHon. ICML
Group A
What has the model
learned?
(holisHc or modular level)
Model
Specific

11. Examples of Post-hoc Explana2ons
11
Rudin, C., and Ertekin, S ̧. 2018. Learning customized and op>mized lists of rules with mathema>cal programming.
Mathema'cal Programming Computa'on 10(4):659–702
Group A
What has the model
learned?
(holis>c or modular level)
Model
Specific

12. Examples of Post-hoc Explana2ons
12
Montavon, G.; Lapuschkin, S.; Binder, A.; Samek, W.; and Mu ̈ller, K.-R. 2017. Explaining nonlinear classificaIon de- cisions
with deep Taylor decomposiIon. Pa#ern Recogni- .on 65:211–222.
Group A
What has the model
learned?
(holisIc or modular level)
Model
Specific

13. Examples of Post-hoc Explana2ons
13
Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016. ”Why Should I Trust You?”: Explaining the PredicJons of Any Classifier.
ACM KDD
Group A
What has the model
learned?
(holisJc or modular level)
Model
AgnosJc

14. Examples of Post-hoc Explana2ons
14Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2018. Anchors: High-Precision Model-Agnostic Explanations. AAAI Press 32:1527–
Group A
What has the model
learned?
(holisOc or modular level)
Model
AgnosOc

15. Examples of Post-hoc Explana2ons
15
Henelius, A.; Puolama ̈ki, K.; and Ukkonen, A. 2017. Interpre?ng Classifiers through ADribute Interac?ons in Datasets.
ICML
Group A
What has the model
learned?
(holis?c or modular level)
Model
Agnos?c

16. General Concepts & Methods
• Rule Sets
• Sensi+vity Analysis
• Induc+ve Logic/Programming
• Recently:
• Counterfactuals
• Adversarial approaches
• Game theory
16

17. Interes'ng Direc'ons
17

18. Interes'ng Direc'ons
18

19. Open Challenges
• No formal agreed upon defini1on
• The no1on of interpretability seems to be an ill-defined term?
• Having agreed upon defini1on avoids reinven1ng the wheel
• Easier to built upon and contribute to prior work
• Rigorous, agreed upon evalua1on metrics
• Clear dis1nc1on of human vs. machine based evalua1on metrics
• Provide a clear picture of what is working and what needs improvement
19

20. Open Challenges
• Stochas(c nature of the models, different random seeds lead to different
outcomes for the same models
• P Henderson, R Islam, P Bachman, J Pineau, Deep Reinforcement Learning That
MaAers, AAAI 20018
• Models are built on assump(ons à = f( )
• When do they break and how?
20

21. Open Challenges
• Humans are great storytellers/story makers
• Memory championship à Method of loci
• Often humans create stories from small indications which rely upon in order to
build explanations
• These explanations might not have any relation with the underlying actual model
• How to avoid specific cognitive biases?
• Framing effect
• Focusing effect
• Illusory correlation 21

22. Open Challenges
22

23. Open Challenges
23

24. Open Challenges
• Saliency maps can be misleading (Olah et al., 2018)
• Models are uncalibrated
• Need for more transparent approaches
• Bringing another to interpret the exisCng 24

25. References
1. Bodenhofer, Ulrich and Bauer, Peter. Towards an Axiomatic Treatment of
Interpretability, Fuzzy Systems, 2000.
2. Olah, Chris and Satyanarayan, Arvind. The Building Blocks of Interpretability, Distill.pub,
2018.
3. Zadrozny, Bianca and Elkan, Charles. Obtaining calibrated probability estimates from
decision trees and naive bayesian classifiers. In ICML, pp. 609–616, 2001.
4. Zadrozny, Bianca and Elkan, Charles. Transforming classifier scores into accurate
multiclass probability estimates. In KDD, pp. 694–699, 2002.
5. Naeini, Mahdi Pakdaman, Cooper, Gregory F, and Hauskrecht, Milos. Obtaining well
calibrated probabilities using bayesian binning. In AAAI, pp. 2901, 2015.
6. Platt, John et al. Probabilistic outputs for support vector machines and comparisons to
regularized likelihood methods. Advances in large margin classifiers, 10(3): 61–74, 1999.
7. Guo, Chuan and Pleiss, Geoff and Sun, Yu and Weinberger, Kilian Q. On Calibration of
Modern Neural Networks. In ICML 2017.
25

26. THANKS!
Ques-ons?
26
Preprint: https://arxiv.org/abs/1904.02495
ioannis.mitros@insight-centre.org