ESSA 2013 MASS Workshop on Model Analysis Tools
Upcoming SlideShare
Loading in...5
×
 

ESSA 2013 MASS Workshop on Model Analysis Tools

on

  • 287 views

In our proposed worksop, we would like to introduce a set of loosely coupled software components intended to help modellers to explore the behaviour of their models either by performing "smart" ...

In our proposed worksop, we would like to introduce a set of loosely coupled software components intended to help modellers to explore the behaviour of their models either by performing "smart" parameter space explorations and/or participatory experiments by executing their models within a web-based environment.
The first tool on which an introductory tutorial is given is called The Model Exploration Module [1] or MEME, a generic tool that enables orchestrating experiments, managing results. MEME supports model analysis over a range of simulation platforms (RepastJ, NetLogo, Mason). It was designed to run large-scale parameter space explorations on grid/cloud systems or sensitivity analysis through statistical methods based on techniques known in the literature as Design of Experiments.
The second framework is called the The Participatory Extension v2.0 [2] or PET v2.0, which is a further developed version of the original PET [3], a robust and generic web framework that allows modellers to extend their models to participatory simulations. It is a web application that incorporates agent-based simulations into a web interface compatible with any of the major web browsers, enabling users to administrate, run and participate in simulations in a way that they are familiar with, applying the mechanisms and practices they use every day while browsing web-pages and using other web-based applications. Applications of PET v2.0 may include online case studies for demonstrative and teaching purposes, or to conduct laboratory experiments for behavioural studies of a model.

Prerequisites: The tutorial has no particular requirements, but experience in implementing agent-based models is an advantage. The frameworks are introduced through a supplied model, no programming is necessary.

[1] Márton Iványi, László Gulyás, Rajmund Bocsi, Gábor Szemes, and Róbert Mészáros: "Model exploration module", In Agent 2007: Complex Interaction and Social Emergence Conference, Evanston, IL, USA, November 2007

[2] Richard Oliver Legendi, László Gulyás, Tamás Máhr, Rajmund Bocsi, Vilmos Kozma, Gábor Ferschl, Peter Rieger and Jakob Grazzini: "A New Set of Tools Supporting Agent-Based Economic Modelling". (Under publication, submitted to the 16th Portuguese Conference on Artificial Intelligence EPIA 2013, SSM – Social Simulation and Modelling Thematic Track)

[3] Ivanyi, Marton, Rajmund Bocsi, Laszlo Gulyas, Vilmos Kozma and Richard Legendi. "The multi-agent simulation suite." In Emergent Agents and Socialities: Social and Organizational Aspects of Intelligence. Papers from the 2007 AAAI Fall Symposium, pp. 57-64. 2007.

Statistics

Views

Total Views
287
Views on SlideShare
287
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    ESSA 2013 MASS Workshop on Model Analysis Tools ESSA 2013 MASS Workshop on Model Analysis Tools Presentation Transcript

    • Discussion - Matthias Meyer Statistical and Behavioural Model Analysis Tools Tamás Máhr, Richard Legéndi, László Gulyás AITIA International, Inc. tmahr@aitia.ai rlegendi@aitia.ai lgulyas@aitia.ai Warsaw, 16th September 2013
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás A short introduction to designing and executing computational experiments (i.e., simulations) Show how to carry out DoE in practice (MEME) Introduce the Participatory Extension (PET) Aims of this talk What are the practical challenges of designing and executing computational experiments? How to implement and execute the DoE approach in practice? How to enable human subjects to participate in simulations?
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Dowload tools http://bit.ly/essa2013
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Overview ABM research process
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Overview ABM research process 5
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Model Exploration (Lorscheid – Heine – Meyer)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás The General Approach Computer simulations are experiments Where the experimenter tries to determine How the systems response (output) depends On controllable factors (parameters) One may also want to do replicates (cf. RNG seeds) System (p1, p2, p3, p4, …) (r1, r2, r3, r4, …)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Model Exploration (Lorscheid – Heine – Meyer)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Model Exploration (Lorscheid – Heine – Meyer)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Designing a simulation experiment (4) Select a Factorial Design
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Designing a simulation experiment (4) Select a Factorial Design Select a design (or fill in the Design Table) We will see a few further desings
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Practical Steps of (6) Performing a Simulation Experiment Set the parameters (factor values) Combinations or levels WHAT to record Variables (time series) Agent variables (changing length!) Derived values (statistics) WHEN to record @end, @timestep, @N timesteps, @condition WHEN to STOP the simulation Fixed number of steps, condition reached, etc. WHERE to execute Local computer, local cluster, grid, cloud (comfort, pricing)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Practical Steps of (6) Performing a Simulation Experiment (cont'd) Collect the results When using more than a single core/computer, result files end up dispersed Assemble the result set The ordering of the records (table rows) could be arbitrary The number of columns may vary in raw output (e.g., when recording raw agent variables) Often one also needs to pre-process the result set Aggregating, Splitting / Slicing Archive the experiment Keep a 'logbook' of your experiments What results came from what experiment, when and with what settings
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Model Exploration How to implement and execute computational experiments? Practicalities Advanced Designs (beyond factorials) Composite Central Box-Behnken Latin HyperCubes „IntelliSweep“ tools Iterative methods Self-guided searches
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Full Factorial Designs Classic parameter sweeps „as we know them”
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Full Factorial Design A design in which every setting of every factor appears with every setting of every other factor A specialized version of the “brute force” strategy Determines the same number of values (“levels”) for each parameter (factor)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Fractional Factorial Designs Full factorial designs may be demanding even with two levels only (k=10, 2k =1024) A fractional factorial experiment is in which only an adequately chosen fraction of the parameter combinations required for the complete factorial experiment is selected to be run Typically, we pick ¼, ½ of the full factorial
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Example: 2-Level Fractional Factorial Experiments with MEME
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Example: 2-Level Fractional Factorial Experiments with MEME
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Central Composite Design 1 The linear fit provided by the 2-level factorial methods may not be enough To build quadratic, or other higher-order models we need new designs
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Central Composite Design 2 A factorial design with added ‘star points’ on the axis of the parameters + a center point The star points can make new extreme values for the parameters (both min and max) The newly added points help to estimate the curvature
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Central Composite Design 3 There are three different types of CCDs: Circumscribed (CCC) Face centered (CCF) Inscribed(CCI) CCC and CCI are rotatable designs, because every design point is at equal distance from the center The variance of the predicted response of a model based on a rotatable design depends only on the distance from the center point
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás An Alternative Choice to Fit Quadratic Responses The Box-Behnken design an independent quadratic design Does not contain an embedded factorial or fractional factorial Treatment combinations are at the midpoints of edges and at the center A sphere that protrudes through each face
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Properties of The Box-Behnken Design Rotatable (or near rotatable) Requires 3 levels of each factor Have limited capability for orthogonal blocking compared to the central composite designs
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Goals and Details of the Box-Behnken Design The design should be sufficient to fit a quadratic model The ratio of the number of experimental points to the number of coefficients in the quadratic model should be reasonable In fact, their designs kept it in the range of 1.5 to 2.6 The estimation variance Should more or less depend only on the distance from the centre This is achieved exactly for the designs with 4 and 7 factors Should not vary too much inside the smallest (hyper)cube containing the experimental points
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Latin Hypercube Designs, 1 A screening method that Easily handles more than 2 levels and uses much less runs than the factorial design LHD designs operate on A subset of the parameter space defined by a single contiguous interval for each dimension (parameter) – a hypercube The subset is defined by giving the low and high values for each tested factor (parameter)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Latin Hypercube Designs, 2 A criterion: non-collapsing design If one of the parameters has (almost) no influence, then two experiments that differ only in this parameter ‘collapse’ They are like measuring the same point twice This is a waste of the resources (in deterministic cases) Therefore, two design points should not share any coordinate values If it is not known a priori, which dimensions are important
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Latin Hypercube Designs, 3 Definition: A d-dimensional grid of n levels in every dimension Each level occurs only once A non-collapsing design
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Latin Hypercube Designs, 4 A desirable property: Space filling When no details on the functional behavior of the response parameters are available, it is important to obtain information from the entire design space The points of the design should be ‘evenly spread’ over the entire hypercube
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Latin Hypercube Designs, 5 A MAXIMIN design is a set of points, such that The separation distance is maximal I.e., the minimal distance among pairs of points Assuming that the samples represent their ‘surroundings’ one wants to make sure that we use our sample points efficiently We maximize the r common radius of spheres around the design points so that they don’t intersect Any distance metric can be used, but L2 (Euclidean) is a common choice
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás A Note on MAXIMIN LHDs Finding a MAXIMIN, non-collapsing design for many dimensions and a high number of levels is very hard Therefore, often pre-calculated designs are used, and/or the MAXIMIN property is only approximated
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Latin Hypercube Designs in MEME The LHD plugin in MEME supports Up to 100 levels and Up to 10 dimensions Uses predefined experiment designs Calculated by heuristic methods to approximate MAXIMIN LHD designs http://www.spacefillingdesigns.nl/maximin/info.html
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Latin Hypercube Designs in MEME
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Latin Hypercube Designs in MEME
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Dynamic “IntelliSweep” methods So far, the entire design was fixed before starting the experiment There was no feedback from the measured responses to the design Various (optimization) methods exist that use a different strategy Hill climbing, simulated annealing, genetic algorithms, etc.
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Iterative Uniform Interpolation 1 IUI is a response analysis method Refines the parameter domain between iterations to achieve better interpolation (of the response value) Examines “interesting” subintervals by dividing them further Deviation from the previously observed (assumed) gradient spans new measurements.
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Iterative Uniform Interpolation 2
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Genetic Algorithm Driven Methods Optimization Genetic algorithm (GA) is a heuristic optimization method F( o1 , …, on ) → max Can be directly used for response analysis If we are not interested in the entire response surface, but only in high/low response values
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás MEME The Model Exploration Module
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás MEME – The Model Exploration Module Part of the Multi-Agent Simulation Suite (MASS) Yet, useful with most major agent-based platforms (Repast J, Repast Sym*, NetLogo, MASON, EMIL-S, FABLES) GOAL: User firendly toolset for ABM Hiding coding / implementational difficulties as much as possible Ease of use for non-technical people MEME is responsible for Design Execution Data collection of computational experiments (i.e., agent-based simulations)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás MEME – Functions Assists the research process from the point when the implementation of the model (or a version of it) is complete until the publication of the collected results Helps configuring the simulation to record the proper variables Data series from program variables or specific statistics of them Offers wizards for a variety of experimental designs Including fractional factorials and more Orchestrates the execution of the experiment on a single computer or on cluster or in the cloud Collects the recorded data in standard data tables
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás MEME – Functions (cont'd) Functions to preview the results perform exploratory analysis, preliminary charting Export options and interfaces To standard popular statistical packages like R, SPS, STATA Optionally, a personal 'laboratory logbook' Archiving and documenting the computational experiments performed by the modeler
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás MEME History (since 2005) 2005 Tool to administer and process simulation results (Repast J & FABLES) 2006 Setting up config files for parameter sweeps (Repast J) 2007 Distributed execution on local clusters and grids (QosCosGrid) 2007 Design of Experiment (Classic Designs) 2008 Multi-Platform Support (EMIL-S, NetLogo, Custom Java, Repast Sym*) 2008 Advanced statistics in recording 2009 Standard Interface for results processing 2009 Advanced DoE designs (freely extensible architecture) 2010 Intellisweep plugins (iterative, self-guiding exploration of param space) 2011 Execution „in the cloud” (http://modelexploration.aitia.ai/) 2012 Support for the MASON simulation package 2013 MEME goes open source
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Design of Experiments in MEME Classic simulation experiments with „parameter files“ Classic DoE tables DoE Wizards Factorials Fractional Factorials More IntelliSweep experiments Iterative methods Self-guided searching of the parameter space Optimization E.g., Genetic Algorithms
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Simulation in the cloud model exploration service http://modelexplortaion.aitia.ai runs RepastJ, NetLogo, Java models (Mason support is comming soon) uses Amazon EC2 user selects the number of machines used
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás MEME as a Personal Laboratory Logbook The more one gets immersed in (computational) experiments, the more results are filling the hard drive „Nothing can be as alien/unknown than your own code – after two months” Same applies to experimental results and settings, let alone charts „Hey, this is a nice chart. I wish I remembered what exact parameter settings I user to run the model with!” Being disciplined always helps, but tools may help being disciplined. MEME stores all result sets in a DB (together with settings) Grouped by model, version and „batch” (enforced) You can add comments, remarks, descriptions to them
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Param Sweeps Param Sweeps Param Sweeps Param Sweeps Results DB Charts Versioning and Merging Filtering, Processing, Restructuring Views Export (Excell, SPSS, etc.) Import (txt, csv, Excell, etc.)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Practical Start MEME Run Simulation... Open the El Farol model Select experimental design Define output Run experiment Analyze results
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás The El Farol bar problem Parameters: number of agents overcrowding treshold memory size number of strategies Agents are researchers (N=100) They visit a popular but small bar in Santa Fé If attendance > 60   (overcrowded) Who hasn’t come   If attendance <= 60   Who hasn’t come   Each day agents decide individually and in the same time
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás The El Farol model MASON implementation of a NetLogo variant Artifical agents: ARMA-based prediction with history Players have two actions: No go / go If   +1 Score! Goal: get max score
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Summary Discussed the technical issues and challenges of executing computational experiments Explained the challenges of applying the DoE approach in practice Introduced MEME as a tool to assist from the point when the implementation of the simulation is complete Discussed the usage of MEME Together with advanced designs for computational experiments
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás PET The Participatory Extension
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás About PET Participatory Extension (PET) Another component in the MASS toolbox http://pet.aitia.ai/ Converts ABM's to Web Simulations Participatory Experiments Laboratory experiments with human subjects Some agents in the simulation are artificial, some others are controlled by human agents
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Let's play! http://demo1.aitia.ai
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Model List Create Experiments Additional Info
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Subject list (joined) Admin tools Admin page
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás In-experiment admin page Player status (moved/waiting)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás In-experiment subject page Status messages
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Post-experiment scores
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás PET software architecture
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás PET features Runs Mason simulations Can be scaled to run on multiple machines Can be used: Locally Laboratory experiments Policy makers (scenario analysis with a proper model) On any webserver to run constantly Gather data (scores from model and all user actions → replay) Dissemination Questionnaire module Verify if subjects understood the rules Software is already in use by Universiteit van Amsterdam
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás PET requirements An ABM model PET is a generic framework currently Mason models are supported
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás PET requirements An ABM model PET is a generic framework currently Mason models are supported Development of a web interface no restricitions on tools (HTML5, Javascript, GWT, …) communication is based on standards (AJAX) config file maps incoming messages to method calls GWT module available
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás PET communication simple messages invoked by user events (button click) and at initialization getters, setters, etc. call model methods call agent methods action messages simulation is stopped when an action method (decision point!) is hit user events can send action messages public void updateAttendance() public void updateAttendance(final boolean attend)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás PET communication triggered messages from simulation to browser call model or agent method configurable triggers userActed (actionId) messageHandled (messageId) newTurn (actionId) simulationEnd userConnected (playerId, agentType) userDisconnected (playerId, agentType) unhandledExceptionCaught (exceptionType)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás PET communication summary broswer – simulation bidirectional communication change / query the state of the model or your agent (simple message) make the simulation move on (action message) have the simulation send data asynchronously (triggered messages)
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Practical download and start Eclipse install the right GWT plugin from marketplace! import the project you downloaded open the GUI class
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Links to Software http://meme.aitia.ai/ http://modelexploration.aitia.ai/ http://mass.aitia.ai http://pet.aitia.ai/
    • Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás Thank you! Questions? {tmahr|rlegendi|lgulyas}@aitia.ai