Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Study and development of methods and tools for testing, validation and verification of multi-agent systems

720 views

Published on

Doctoral thesis presentation

Published in: Science
  • Be the first to comment

Study and development of methods and tools for testing, validation and verification of multi-agent systems

  1. 1. Study and development of methods and tools for testing, validation and verification of multi-agent systems University of Murcia Department of Information and Communication Engineering Author: Emilio Serrano Fernández Supervisors: Juan Antonio Botía Blaya José Manuel Cadenas Figueredo
  2. 2. Outline 1 2 3 4 5 2 Motivation Quantitative analysis for general MASs Qualitative analysis for MASs with protocols semantically annotated Analysis for MABSs used as model of AmI systems Conclusions and future lines
  3. 3. Outline 1 2 3 4 5 3 Motivation Quantitative analysis for general MASs Qualitative analysis for MASs with protocols semantically annotated Analysis for MABSs used as model of AmI systems Conclusions and future lines
  4. 4. ● Software is used for more life-critical applications every year. ● Bugs cost the U.S. economy an estimated 59.5$ billion annually. ● Bugs must be found by testing, verification, validation and debugging. ● The most common debugging strategy is by brute force. ● Multi-agent systems (MASs): kind of distributed system ● Interaction among agents is performed by complex dialogues. ● Debugging a MAS in which intelligent or emergent behaviors may appear is much more complex. ● This thesis deals with the analysis of interactions in large-scale MASs. Motivation 4 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  5. 5. ● Categorized in the forensic analysis field ● Providing a series of simple displays ● Defining and testing protocols automatically ● Discovering knowledge in the event log ● Shortcomings which motivate this thesis and goals ● Order of the logged events ● Abstract of the presented information ● Detection of undesired emergent behaviors ● Usability ● Analyzing semantics Three main MAS analysis approaches 5 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  6. 6. ● General large-scale FIPA MASs ● Very few elements are assumed. ● MASs with protocols semantically annotated. ● Annotations allows to study semantics. ● Multi-agent based simulations (MABS) which model Ambient Intelligence (AmI) applications. ● Why such a concrete system? Types of MAS studied 6 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  7. 7. Outline 1 2 3 4 5 7 Motivation Quantitative analysis for general MASs Qualitative analysis for MASs with protocols semantically annotated Analysis for MABSs used as model of AmI systems Conclusions and future lines
  8. 8. Abstract of the thesis 8 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  9. 9. ● General approach to provide forensic analysis of runs ● Capturing, ordering, and representation of messages ● Capturing: Aspect oriented programming (AOP) is employed. ● Relational data base (RDB) stores the messages. ● Transparent to developer ● Ordering: The global state is distributed and that a common time base does not exist ● Vector clocks: an array of n integers in a MAS with n agents ● m1_ v1 happened before m2_ v2 if v1 < v2 ● Management required → AOP Infrastructure for Forensic Analysis of Multi-Agent Systems 9 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  10. 10. ● They help to find typical bugs: ● Uninitialized agent, failure to send, wrong recipient, message sent multiple times, and wrong message sent. ● Sequence diagrams are very popular ● There is an information loss. ● The use of order graphs is proposed ● Concurrent events are clearly shown. ● Acyclic directed graph OG(V,E), where V are the selected messages and E the transitivity reduction of the less-strict relation between vector clocks. Studying messages via simple displays 10 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  11. 11. Order graph for case study, fire example 11 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  12. 12. ● When the number of messages and agents grows, the utility of the visual representation decreases. ● Abstract graphs and collaboration graphs Summarizing the order graphs 12 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  13. 13. ● How to detect undesired emergent behaviors? ● Database M with messages → Data mining ● Fields: sender, receivers, performative and content. ● Patterns can be detected ● Association rules and Clustering Intelligent Data Analysis over the Database for Forensic Analysis 13 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  14. 14. ● Association rules: technique to discover relations between variables in large databases ● Example: at1=v1 and at2=v2 then at3=v3 and at4=v4 (0.5) ● Relations between fields in the messages ● Goals: bugs detection and automatic testing ● MAS typically has a non-deterministic behavior Discovering relationships among attributes by association rules 14 PERFORMATIVE=CFP ==> SENDER=client conf:(1) PERFORMATIVE=ACCEPTPROPOSAL ==> SENDER=client conf:(1) PERFORMATIVE=REJECTPROPOSAL ==> SENDER=client conf:(1) PERFORMATIVE=PROPOSE ==> RECEIVER=client conf:(1) PERFORMATIVE=INFORM ==> RECEIVER=client conf:(1) ... Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  15. 15. ● Discovering groups of agents which behave similarly ● Similarity → agents which communicate with same agents ● ROCK algorithm is applied → categorical data ● M in the form of messages is not valid anymore ● Two agents are neighbors if they communicate with the same agent ● Clusters → agents which share a high number of neighbors. Discovering agents behaving similarly by the ROCK clustering algorithm 15 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  16. 16. ● Discovering agents which collaborate ● Collaboration → large interchange of messages ● The k-means algorithm is applied → vectors of real numbers ● Vectors must reflect the communication activity ● Groups are formed with agents which had a high communication activity among them Discovering agents interacting together by the k-means clustering algorithm 16 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  17. 17. Collapsed similarity and collaboration graphs for a case study, cinema example 17 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  18. 18. ● Clustering approach presents some limitations ● Intra and inter cluster visual information ● Information about the most important elements ● Amount of knowledge required by the user ● A solution is the use of techniques based on social networks analysis (SNA) ● Collaboration graph → social network ● Pathfinder algorithm generates pathfinder networks (PFNETs) ● They allow to reveal the underlying organization of a system ● P.e. psychology and scientometrics ● It keeps only those links whose weights do not violate the triangle inequality Analysis by Pathfinder Networks, Improving the Clustering Approach 18 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  19. 19. ● A PFNET(r, q) is a septuple (N, E, W, LLR, LMR, r, q) ● Only (A, E, W) is considered ● Similarity PFNET ● W function bounded by (0,1] ● 1 if agents interact with same agents using same number of messages ● Collaboration PFNET ● W function bounded by [0,1) ● 0 if agents do not interact at all Obtaining Pathfinder Networks for agents 19 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  20. 20. Collaboration PFNET for a case study, cinema example 20 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  21. 21. Implementation, ACLAnalyser 21 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions Presentation: videosACLAnalyser.mp4 Project website: aclanalyser.sourceforge.net/
  22. 22. ● Quantitative analysis for general large-scale MASs ● General infrastructure for forensic analysis ● AOP, RDB, vector clocks ● Simple displays for MASs runs. ● Order graphs, sequences diagrams, abstract graphs, collaboration graphs ● Data mining ● Apriori, ROCKS, K-means ● Social network analysis ● Pathfinder → PFNETs (similarity and collaboration) ● Implementation ● ACLAnalyser Contributions of this part 22 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  23. 23. Outline 1 2 3 4 5 23 Motivation Quantitative analysis for general MASs Qualitative analysis for MASs with protocols semantically annotated Analysis for MABSs used as model of AmI systems Conclusions and future lines
  24. 24. Abstract of the thesis 24 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  25. 25. ● Qualitative approach ● The target MAS cannot be general → Requirements to capture semantics ● Protocols specification ● Protocol results + semantics + data mining → context models ● Logical theories that capture regularities in previously observed interactions ● Utility? ● Summary ● Predictions, infer the mental states definitions, trustworthiness Qualitative Analysis from Interactions, Using Protocols Semantically Annotated 25 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  26. 26. Protocols Semantically Annotated, a negotiation 26 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions Protocol paths + ground terms = context = tuples for data mining
  27. 27. ● Data mining algorithms usually assume a fixed number of attributes ● Strategies for preprocessing ● Different paths → Different messages/constraints/variables ● Different data set for each observed path ● Using all messages/constraints and “?”· ● Iterations → Several constants g1,g2...gn ● N “copies” of each variable can be kept ● Considering only the first/last ground term g1/gn ● Class ● Protocol result/Constraint result Mining protocol executions: Dealing with paths, loops, and variables 27 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  28. 28. Results for a case study, car negotiation 28 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  29. 29. Implementation, Protocol miner 29 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  30. 30. ● Qualitative analysis for MASs with protocols semantically annotated ● Context models = data mining over paths and semantics ● Formal framework for protocols ● Strategies for the preprocess ● Experimental results ● Implementation Contributions of this part 30 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  31. 31. Outline 1 2 3 4 5 31 Motivation Quantitative analysis for general MASs Qualitative analysis for MASs with protocols semantically annotated Analysis for MABSs used as model of AmI systems Conclusions and future lines
  32. 32. Abstract of the thesis 32 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  33. 33. ● Multi-agent based simulations (MABS) ● Ideal technology for the validation of AmI applications. ● Ambient Intelligence (AmI) ● Applications are endless ● Smart homes, health monitoring and assistance, hospitals, transportation, emergency services, education, workplaces, etc. ● Direct validation may be impractical. ● Methodology for the development and analysis of MABSs to validate AmI applications. ● MABSs are complex → forensic analysis and new techniques. Forensic Analysis of MABSs as Tool to Develop AmI Systems 33 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  34. 34. 34 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions AmI Systems
  35. 35. AVA, An Agent based methodology for the Validation of AmI systems 35 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  36. 36. Forensic analysis infrastructure for MABS 36 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  37. 37. Results for a case study, UbikSim 37 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  38. 38. Implementation, UbikSim 38 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions First release: videosUbikSimHD.mp4 Current version: videosUbikSim-office.avi Project website: ubiksim.sourceforge.net/
  39. 39. ●AVA methodology ●Analysis of MABSs which model AmI systems ●Simpler simulations ●Reality injection ●Forensic analysis ●Implementation, UbikSim Contributions of this part 39 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  40. 40. Outline 1 2 3 4 5 40 Motivation Quantitative analysis for general MASs Qualitative analysis for MASs with protocols semantically annotated Analysis for MABSs used as model of AmI systems Conclusions and future lines
  41. 41. ● Contribution: advancing the state of the art in the analysis of multi-agent systems (MASs) interactions. ● Based on a forensic analysis. ● Very few elements that can be assumed for the analysis of a general MAS interactions. ● Database with messages. ● An infrastructure for forensic analysis has been provided. ● AOP, RDB, Vector clocks, Order graphs, Sequences diagrams, Collaboration graphs, and Abstract graphs. ● Interaction among agents can generate unwanted emergent behaviors. ● Learning techniques can be applied: association rules and clustering. Conclusions I 41 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  42. 42. ● Human intervention is required to assess the results of data mining ● Pathfinder networks (PFNETs) ● Clues about unwanted behaviors which must be confirmed ● Restrictions on the MAS design allows a more a powerful analysis ● Protocols semantically annotated allow to study not only the “envelope” ● Definition, creation, and analysis of Multi-agent based simulations (MABSs) used as model to validate Ambient Intelligence (AmI) systems ● Testing AmI complex and costly → MABS are also complex systems → forensic analysis methods and tools ● We have to keep working Conclusions II 42 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  43. 43. ● Quantitative analysis ● Studying new aspects in agent societies to be represented. ● Studying alternative techniques. ● Improving the degree of automation. ● Qualitative analysis ● Using more real-world examples. ● Studying the performance for different strategies. ● Employing more advanced machine learning methods. ● Analysis for MABS ● Automatic construction and verification of the environment model. ● Using learning agents. ● More cases study. Future lines 43 Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions
  44. 44. Journals • Validating Ambient Intelligence based Ubiquitous Computing Systems by Means of Artificial Societies. Emilio Serrano and Juan Botía. Information Sciences. In press (Impact Factor 2010, 2.833, Q1) • Debugging complex software systems by means of pathfinder networks. Emilio Serrano, Arnaud Quirin, Juan Botia and Oscar Cordón. Information Sciences 180 (5) (2010) 561-- 583. (Impact Factor 2009, 3.291, Q1) . • Ubik: a multi-agent based simulator for ubiquitous computing applications. Emilio Serrano, Juan A. Botia, and Jose M. Cadenas. Journal of Physical Agents, 3(2), 2009. Indexed in Scopus. • Intelligent Data Analysis applied to Debug Complex Software Systems. Emilio Serrano, Jorge J. Gomez-Sanz, Juan A. Botia, Juan Pavon. Neurocomputing, 72(13-15):2785 -- 2795, 2009. (Impact Factor 2009, 1.440, Q2). Relevant Publications 44
  45. 45. Conferences • Mining Qualitative Context Models from Multiagent Interactions. Emilio Serrano, Michael Rovatsos and Juan Botía. Extended Abstract in AAMAS2011. • Human behaviours simulation in ubiquitous computing environments. Teresa Garcia-Valverde, Francisco Campuzano, Emilio Serrano and Juan A. Botia. Workshop on Multi-Agent Systems and Simulation (MAS&S) at MALLOW10. • Social simulation for AmI systems engineering. Teresa Garcia-Valverde, Emilio Serrano and Juan A. Botia. International Conference on Hybrid Artificial Intelligence Systems (HAIS'10), 2010. • Incremental deployment of large-scale AmI system by means of social models. Juan A. Botia, Emilio Serrano, Teresa Garcia-Valverde and Antonio Gomez-Skarmeta. Proceedings of International Workshop on ``Simulation of Complex Social Systems'' (SiCoSSys 2009). • Artificial societies immersed in an Ambient Intelligence Environment. Teresa Garcia-Valverde, Emilio Serrano, Juan A. Botia, Antonio Gomez-Skarmeta and Jose M. Cadenas. Proceedings of The 1st Workshop on Social Simulation on International Joint Conferences on Artificial Intelligence, 2009. • Infrastructure for forensic analysis of multi-agent based simulations. Emilio Serrano, Juan A. Botía Blaya and Jose M. Cadenas. Seventh international Workshop on Programming Multi-Agent Systems. Promas 2009 • Construction and debugging of a multi-agent based simulation to study ambient intelligence applications. Emilio Serrano, Juan A. Botía Blaya, and Jose M. Cadenas. International Work-Conference on Artificial Neural Networks (IWANN2009). • Infrastructure for forensic analysis of multi-agent systems. Emilio Serrano and Juan Botia. In Programming Multi- Agent Systems: 6th International Workshop, PROMAS 2008. • Testing and debugging of MAS interactions with INGENIAS. Jorge J. Gómez-Sanz, Juan Botia, Emilio Serrano, and Juan Pavón. In Agent-Oriented Software Engineering IX: 9th International Workshop, AOSE 2008 Relevant Publications II 45
  46. 46. • Jorge J. Gómez-Sanz and Juan Pavón • Complutense University of Madrid • Oscar Cordón and Arnaud Quirin • European Centre for Soft Computing • Michael Rovatsos • University of Edinburgh • Teresa Garcia-Valverde, Francisco Campuzano, and Andrés Muñoz • My group • Acknowledgements: Thesis supported by the Spanish Ministry of Science and Innovation under the grant AP2007-04080 of the FPU program, and in the scope of the Research Projects CARONTE, DIA++ and through the Fundación Séneca within the Program “Generación del Conocimiento Científico de Excelencia” Collaborators and acknowledgements 46
  47. 47. University of Murcia Department of Information and Communication Engineering Thank you very much for your attention "Science is forever a search, never really a finding. It is a journey, never really an arrival" Karl Popper 47

×