Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Towards XMAS: eXplainability through Multi-Agent Systems

In the context of the Internet of Things (IoT), intelligent systems (IS) are increasingly relying on Machine Learning (ML) techniques. Given the opaqueness of most ML techniques, however, humans have to rely on their intuition to fully understand the IS outcomes: helping them is the target of eXplainable Artificial Intelligence (XAI). Current solutions – mostly too specific, and simply aimed at making ML easier to interpret – cannot satisfy the needs of IoT, characterised by heterogeneous stimuli, devices, and data-types concurring in the composition of complex information structures. Moreover, Multi-Agent Systems (MAS) achievements and advancements are most often ignored, even when they could bring about key features like explainability and trustworthiness. Accordingly, in this paper we (i) elicit and discuss the most significant issues affecting modern IS, and (ii) devise the main elements and related interconnections paving the way towards reconciling interpretable and explainable IS using MAS.

  • Login to see the comments

  • Be the first to like this

Towards XMAS: eXplainability through Multi-Agent Systems

  1. 1. Towards XMAS: eXplainability through Multi-Agent Systems Giovanni Ciatto∗ Roberta Calegari∗ Andrea Omicini∗ Davide Calvaresi† ∗Dipartimento di Informatica – Scienza e Ingegneria (DISI) Alma Mater Studiorum – Universit`a di Bologna {giovanni.ciatto , roberta.calegari, andrea.omicini}@unibo.it †University of Applied Sciences and Arts Western Switzerland davide.calvaresi@hevs.ch 1st Workshop on Artificial Intelligence & Internet of Things Rende, Italy – November 21, 2019 Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 1 / 17
  2. 2. Motivation & Context Next in Line. . . 1 Motivation & Context 2 State of the art 3 eXplainability through Multi-Agent Systems 4 Conclusions Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 1 / 17
  3. 3. Motivation & Context Context Some well known facts: Pervasive adoption of AI- and ML-powered IoT solutions world-wide for automation, monitoring, and decision support ⇒ Several activities are (partially?) delegated to intelligent machines ! even activities from critical domains: finance, healthcare, etc Especially in ML, we let machines learn specific tasks from data through the production of numeric predictors, a.k.a. black-boxes instead of programming such tasks ourselves Unfortunately, black-boxes tend to be inherently opaque w.r.t. the knowledge they acquire from data [12] sub-optimal in performance as they are trained to minimise errors Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 2 / 17
  4. 4. Motivation & Context Motivation Opaqueness of ML-based predictors brings several drawbacks [9, 12]: difficulty in understanding what a black-box has learned from data e.g. “snowy background” problem [16] difficulty in spotting “bugs” in what a numeric predictor has learned because such knowledge is not explicitly represented several failures of ML-based systems reported so far e.g. black people classified as gorillas [6] e.g. wolves classified because of snowy background [16] e.g. unfair decisions in automated legal systems [20] lawmakers recognised citizens’ right to meaningful explanations [18] about the logic behind automated decision making e.g. in General Data Protection Regulation (GDPR) [8] Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 3 / 17
  5. 5. Motivation & Context The problem with ML-based AI Trustworthiness How can we trust machines we do not fully control? ↓ Controllability How can we control machines we do not fully understand? ↓ Understandability How can we understand distributed, numeric representations of knowledge? Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 4 / 17
  6. 6. Motivation & Context The problem with ML-based IoIT Other issues, made evident by IoIT: Lack of (full) automation Training of ML predictors heavily depends on the experience of human data scientists Centralisation of data & computation Datasets cannot be easily moved & training can hardly be distributed Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 5 / 17
  7. 7. State of the art Next in Line. . . 1 Motivation & Context 2 State of the art 3 eXplainability through Multi-Agent Systems 4 Conclusions Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 5 / 17
  8. 8. State of the art The eXplanable AI (XAI) approach [10] I The XAI community is nowadays facing such understandability issues Focus on techniques easing the interpretation of numeric predictors a.k.a. “opening the black box”, or look into it [9] From [12] In particular, most efforts are devoted to: specific sorts of tasks, e.g. classification and regression specific sorts of data, e.g. images, text, or tables specific sorts of predictors, e.g. neural networks, SVM Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 6 / 17
  9. 9. State of the art The eXplanable AI (XAI) approach [10] II Studying techniques such as saliency maps [5], feature importance [19], sensitivity analysis [13], activation maximisation [22] from [16] Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 7 / 17
  10. 10. State of the art Symbolic vs Numeric AI ML is strictly a subset of AI Several approaches lay under the Symbolic AI umbrella often employed in expert, decision-support, or recommendation systems There, knowledge is represented through symbolic languages, in the form of logic rules or facts less prone to opacity issues both machine- and human-interpretable Main drawbacks: less flexibility w.r.t. numeric approaches symbolic knowledge is mostly handcrafted Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 8 / 17
  11. 11. State of the art About Symbolic AI Symbolic AI is largely employed in well established research areas, such as: Logic Programming (LP) [2] Studying how symbolic rules may be employed as a programming language Multi Agents Systems (MAS) [21] Studying complex systems composed by several autonomous and interacting entities called agents, reasoning or planning through LP Argumentation [11] Studying how agents may debate with each others in spite of opposing or contradictory points of view on some subject—or learn from each others Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 9 / 17
  12. 12. State of the art Symbolic Knowledge Extraction (SKE) Symbolic and numeric approaches to AI are not competing anymore conversely, they are complementary to each others SKE is the bridge between the two worlds Several works have been proposed into the literature concerning SKE describing methods to extract decision rules/trees from black-boxes most of which surveyed in [1, 9] Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 10 / 17
  13. 13. eXplainability through Multi-Agent Systems Next in Line. . . 1 Motivation & Context 2 State of the art 3 eXplainability through Multi-Agent Systems 4 Conclusions Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 10 / 17
  14. 14. eXplainability through Multi-Agent Systems Interpretation vs Explanation Such terms are wrongly considered synonyms [12, 17] We thus adopt the following conceptual framework: interpretation def = the cognitive activity of binding symbols/numbers to their meaning explanation def = the social activity of easing someone’s interpretation e.g. by providing examples, or background knowledge Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 11 / 17
  15. 15. eXplainability through Multi-Agent Systems XMAS Vision We re-interpret ML-based systems as MAS where: SKE ML SKE ML Loan? Debate No Why? Debate Income < 1.500 € Example ForIncome<1500€ & Loan Debate Permanent Job Assuming several data-sets exist Agents wrap a black-box trained on a data-set Agents extract rules from black-boxes Debating protocols are employed by agents to: compute decisions explain decisions Perfect metaphor for IoIT Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 12 / 17
  16. 16. eXplainability through Multi-Agent Systems XMAS Vision – Multiple Expected Advantages Explanations are interactive in nature Multiple agents ↔ multiple perspectives similarly to ensemble techniques Symbols are a lingua franca for knowledge (sharing) predictions / explanations from different predictors can be combined Symbolic, aggregated knowledge could be moved among agents even when the original datasets cannot → thus improving distribution while preserving privacy The future: agents teaching to each others, through explanations by exchanging symbolic knowledge → thus improving automation in training Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 13 / 17
  17. 17. eXplainability through Multi-Agent Systems Paper contribution i∗ modelling of this research line describing the foreseeable goals & activities . . . and their dependencies Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 14 / 17
  18. 18. Conclusions Next in Line. . . 1 Motivation & Context 2 State of the art 3 eXplainability through Multi-Agent Systems 4 Conclusions Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 14 / 17
  19. 19. Conclusions Summing up ML-powered AI is everywhere but it not the silver-bullet Increasing demand of explanabilty for ML-based systems XAI mostly focus on interpretability, a.k.a. opening the black-boxes whereas explanabilty requires interaction Idea: extract symbolic knowledge from black-boxes and use debates to explain it This is expected to bring several benefits, even beyond interpretability Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 15 / 17
  20. 20. Conclusions Future Works Comparison, assessment, and generalisation of SKE algorithms development of software libraries for SKE e.g. extending Sci-Kit Learn [14] Technological integration of SKE with symbolic frameworks e.g. the tuProlog engine [7] Development, validation, and simulation of debating protocols development of simulation facilities e.g. extending the Alchemist meta-simulator [15] development enabling infrastructures for real-world experiments e.g. extending the TuSoW technology [4] e.g. robust & trustworthy through Blockchain technologies [3] Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 16 / 17
  21. 21. Towards XMAS: eXplainability through Multi-Agent Systems Giovanni Ciatto∗ Roberta Calegari∗ Andrea Omicini∗ Davide Calvaresi† ∗Dipartimento di Informatica – Scienza e Ingegneria (DISI) Alma Mater Studiorum – Universit`a di Bologna {giovanni.ciatto , roberta.calegari, andrea.omicini}@unibo.it †University of Applied Sciences and Arts Western Switzerland davide.calvaresi@hevs.ch 1st Workshop on Artificial Intelligence & Internet of Things Rende, Italy – November 21, 2019 Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 17 / 17
  22. 22. Bibliography References I Robert Andrews, Joachim Diederich, and Alan B. Tickle. Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge-Based Systems, 8(6):373–389, December 1995. Krzysztof R. Apt. The logic programming paradigm and prolog. CoRR, cs.PL/0107013, 2001. Giovanni Ciatto, Stefano Mariani, and Andrea Omicini. Blockchain for trustworthy coordination: A first study with Linda and Ethereum. In 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pages 696–703, December 2018. Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 17 / 17
  23. 23. Bibliography References II Giovanni Ciatto, Lorenzo Rizzato, Andrea Omicini, and Stefano Mariani. Tusow: Tuple spaces for edge computing. In The 28th International Conference on Computer Communications and Networks (ICCCN 2019), Valencia, Spain, August 2019. IEEE. R. Cong, J. Lei, H. Fu, M. Cheng, W. Lin, and Q. Huang. Review of visual saliency detection with comprehensive information. IEEE Transactions on Circuits and Systems for Video Technology, 29(10):2941–2959, Oct 2019. Kate Crawford. Artificial intelligence’s white guy problem. The New York Times, 25, 2016. Enrico Denti, Andrea Omicini, and Roberta Calegari. tuProlog: Making Prolog ubiquitous. ALP Newsletter, October 2013. Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 17 / 17
  24. 24. Bibliography References III General Data Protection Regulation (GDPR). Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32016R0679. Online; accessed on October 11, 2019. Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and Fosca Giannotti. A survey of methods for explaining black box models. CoRR, abs/1802.01933, 2018. David Gunning. Explainable artificial intelligence (XAI). Funding Program DARPA-BAA-16-53, DARPA, 2016. Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 17 / 17
  25. 25. Bibliography References IV Dionysios Kontarinis. Debate in a multi-agent system : multiparty argumentation protocols. PhD thesis, Universit´e Ren´e Descartes, Paris V, 2014. https://tel.archives-ouvertes.fr/tel-01345797. Zachary Chase Lipton. The mythos of model interpretability. CoRR, abs/1606.03490, 2016. Julian D Olden and Donald A Jackson. Illuminating the “black box”: a randomization approach for understanding variable contributions in artificial neural networks. Ecological Modelling, 154(1):135 – 150, 2002. Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 17 / 17
  26. 26. Bibliography References V F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. Danilo Pianini, Sara Montagna, and Mirko Viroli. Chemical-oriented simulation of computational systems with ALCHEMIST. Journal of Simulation, 2013. Marco T´ulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should I trust you? Explaining the predictions of any classifier. CoRR, abs/1602.04938, 2016. Avi Rosenfeld and Ariella Richardson. Explainability in human–agent systems. Autonomous Agents and Multi-Agent Systems, may 2019. Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 17 / 17
  27. 27. Bibliography References VI Andrew D Selbst and Julia Powles. Meaningful information and the right to explanation. International Data Privacy Law, 7(4):233–242, 12 2017. Marina M.-C. Vidovic, Nico G¨ornitz, Klaus-Robert M¨uller, and Marius Kloft. Feature importance measure for non-linear learning algorithms. CoRR, abs/1611.07567, 2016. Rebecca Wexler. When a computer program keeps you in jail: How computers are harming criminal justice. New York Times, 2017. Michael Wooldridge. An Introduction to MultiAgent Systems. Wiley Publishing, 2nd edition, 2009. Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 17 / 17
  28. 28. Bibliography References VII Luisa M. Zintgraf, Taco Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. ArXiv, abs/1702.04595, 2017. Ciatto et al. (UNIBO, HES-SO) Towards XMAS AI&IoT – Nov 21, 2019 17 / 17

×