Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

An Abstract Framework for Agent-Based Explanations in AI

We propose an abstract framework for XAI based on MAS encompassing the main definitions and results from the literature, focussing on the key notions of interpretation and explanation.

  • Be the first to comment

  • Be the first to like this

An Abstract Framework for Agent-Based Explanations in AI

  1. 1. An Abstract Framework for Agent-Based Explanations in AI Giovanni Ciatto∗ Davide Calvaresi† Michael I. Schumacher† Andrea Omicini∗ ∗Dipartimento di Informatica – Scienza e Ingegneria (DISI) Alma Mater Studiorum – Universit`a di Bologna {giovanni.ciatto , andrea.omicini}@unibo.it †University of Applied Sciences and Arts Western Switzerland {davide.calvaresi, michael.schumacher}@hevs.ch International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) May 9 – 13, 2020, Auckland, New Zeland Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 1 / 17
  2. 2. Motivation & Context Next in Line. . . 1 Motivation & Context 2 Fundamentals 3 Understandability in data-driven intelligent systems 4 Conclusions Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 1 / 17
  3. 3. Motivation & Context Context Some well known facts: Increasing adoption of autonomous intelligent systems for automation, monitoring, and decision support ⇒ Increasing amounts of activities delegated to autonomous agents ! even critical ones, e.g., finance, healthcare, etc Increasing exploitation of ML to let agents learn tasks from data alternative to manual programming Involving black-box predictors which are inherently opaque Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 2 / 17
  4. 4. Motivation & Context Motivation Opaqueness of ML-based predictors brings several drawbacks [5, 7]: difficulty in understanding what agents learn from data e.g. “snowy background” problem [9] difficulty in spotting “bugs” w.r.t. expected behaviour because such knowledge is not explicitly represented several failures of ML-based systems reported so far [2, 9, 11] lawmakers recognised citizens’ right to meaningful explanations [10] about the logic behind automated decision making e.g. in General Data Protection Regulation (GDPR) [4] =⇒ need to make AI more understandable [6] understandable → control / robustness → trust Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 3 / 17
  5. 5. Motivation & Context The eXplanable AI (XAI) approach [6] The XAI community is nowadays facing such understandability issues Focus on techniques easing the interpretation of numeric predictors a.k.a. “opening the black box”, or look into it [5] From [7] In particular, most efforts are devoted to: specific sorts of tasks, e.g. classification and regression specific sorts of data, e.g. images, text, or tables specific sorts of predictors, e.g. neural networks, SVM Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 4 / 17
  6. 6. Motivation & Context Contribution of the paper Discussions in the field of XAI are often ambigous Due to the strong reliance on informal notions such as explanation, interpretation, or understandability Which are often used interchangeably In this work We provide a clear unambiguous definition of two fundamental notions: explanation interpretation proposing an abstract framework leveraging on the MAS background Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 5 / 17
  7. 7. Fundamentals Next in Line. . . 1 Motivation & Context 2 Fundamentals 3 Understandability in data-driven intelligent systems 4 Conclusions Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 5 / 17
  8. 8. Fundamentals About interpretation Definition: interpretability A fuzzy and subjective property any object X may satisfy into some agent A’s perspective interpretability of some object X is not an absolute property it only makes sense in presence of an observer A we model interpretation as an observer-specific function IA(X) → [0, 1] the particular value of IA for some X is not that relevant as long as comparisons are possible: IA(X) > IA(X ) ≥ IA(X ) . . . Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 6 / 17
  9. 9. Fundamentals About explanation Definition: explanation An objective activity any agent may perform to make an object X more interpretable explaining an object X = searching for another object X s.t. X is more interpretable than X, and X has an high fidelity w.r.t. X we model explanation as a function E(X) → X whereas difference in fidelity is measured through a function ∆f (X, X ) → [0, ∞) Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 7 / 17
  10. 10. Fundamentals Understandability in a nutshell Definition: understandability The soft goal pursued by an agent A willing to make some object X interpretable to some observer B, by looking for the right explanation Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 8 / 17
  11. 11. Understandability in data-driven intelligent systems Next in Line. . . 1 Motivation & Context 2 Fundamentals 3 Understandability in data-driven intelligent systems 4 Conclusions Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 8 / 17
  12. 12. Understandability in data-driven intelligent systems ML-based intelligent systems A common situation for intelligent agents is to leverage on ML to learn tasks from data This implies a predictor M to be trained on some dataset (X, Y ) For any given task, many families of predictors may be suitable eg neural networks, SVM, decision trees, linear models, etc. In particular, training aims at selecting the best predictor M w.r.t. some predictive performance measure of choice the data at hand Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 9 / 17
  13. 13. Understandability in data-driven intelligent systems The role of representations in ML However, interpretability of predictors is an important feature as well Predictors, as abstract objects, are not directly interpretable Definition: Predictor representations A predictor M may have one or more representation R = r(X, M), describing its behaviour for some input data X eg heatmaps, feature importance vectors, decision boundary plots, etc ! representations are actually interpretable by observers Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 10 / 17
  14. 14. Understandability in data-driven intelligent systems Global vs. local representation Local representations Describe a predictor behaviour w.r.t. some portion of the input space (e.g. 1 instance) Global representations Describe a predictor behaviour w.r.t. the whole input space Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 11 / 17
  15. 15. Understandability in data-driven intelligent systems Representations interpretability Not all representations are equally interpretable Nor can a representation fit all possible cases eg heatmaps are better suited for image classifiers Predictor families come with some natural representation some are considered more interpretable than others Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 12 / 17
  16. 16. Understandability in data-driven intelligent systems Representations vs Explanation To make some predictor M more interpretable for an agent A, one may either: change representation, or search for a better explanation M = E(M) The latter case make sense if M has an high fidelity to M for some input data X r(X, M ) is more interpretable than any other r(X, M) Takeaway Explaining a black-box predictor is about searching approximate models amenable of more interpretable representations Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 13 / 17
  17. 17. Understandability in data-driven intelligent systems Example: Symbolic Knowledge Extraction (SKE) Neural network explanation −−−−−−→ Decision tree/rules eg symbolic knowledge extraction out of neural networks [1, 5] Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 14 / 17
  18. 18. Conclusions Next in Line. . . 1 Motivation & Context 2 Fundamentals 3 Understandability in data-driven intelligent systems 4 Conclusions Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 14 / 17
  19. 19. Conclusions Summing up ML-powered AI is everywhere but it not the silver-bullet Increasing demand of understandability for ML-based systems XAI mostly focus on building more interpretable representation a.k.a. opening the black-boxes [5] Most discussions are imprecise as they leverage on ambiguous notions and terms → Abstract framework deeply rooted in the MAS, to properly define interpretation and explanation Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 15 / 17
  20. 20. Conclusions Future Works Extension of the conceptual framework towards the multi-agent case user-2-agents and agent-2-agents cases Design, development, and validation of protocols for cooperative/competitive best explanation search Comparison, assessment, and generalisation of SKE algorithms development of software libraries for SKE e.g. extending Sci-Kit Learn [8] Technological integration of SKE with symbolic frameworks e.g. the tuProlog engine [3] Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 16 / 17
  21. 21. An Abstract Framework for Agent-Based Explanations in AI Giovanni Ciatto∗ Davide Calvaresi† Michael I. Schumacher† Andrea Omicini∗ ∗Dipartimento di Informatica – Scienza e Ingegneria (DISI) Alma Mater Studiorum – Universit`a di Bologna {giovanni.ciatto , andrea.omicini}@unibo.it †University of Applied Sciences and Arts Western Switzerland {davide.calvaresi, michael.schumacher}@hevs.ch International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) May 9 – 13, 2020, Auckland, New Zeland Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 17 / 17
  22. 22. Bibliography References I [1] Robert Andrews, Joachim Diederich, and Alan B. Tickle. Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge-Based Systems, 8(6):373–389, December 1995. [2] Kate Crawford. Artificial intelligence’s white guy problem. The New York Times, 25, 2016. [3] Enrico Denti, Andrea Omicini, and Roberta Calegari. tuProlog: Making Prolog ubiquitous. ALP Newsletter, October 2013. Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020
  23. 23. Bibliography References II [4] General Data Protection Regulation (GDPR). Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32016R0679. Online; accessed on October 11, 2019. [5] Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and Fosca Giannotti. A survey of methods for explaining black box models. CoRR, abs/1802.01933, 2018. [6] David Gunning. Explainable artificial intelligence (XAI). Funding Program DARPA-BAA-16-53, DARPA, 2016. Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020
  24. 24. Bibliography References III [7] Zachary Chase Lipton. The mythos of model interpretability. CoRR, abs/1606.03490, 2016. [8] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. [9] Marco T´ulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should I trust you? Explaining the predictions of any classifier. CoRR, abs/1602.04938, 2016. [10] Andrew D Selbst and Julia Powles. Meaningful information and the right to explanation. International Data Privacy Law, 7(4):233–242, 12 2017. Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020
  25. 25. Bibliography References IV [11] Rebecca Wexler. When a computer program keeps you in jail: How computers are harming criminal justice. New York Times, 2017. Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020

×