In our proposed worksop, we would like to introduce a set of loosely coupled software components intended to help modellers to explore the behaviour of their models either by performing "smart" parameter space explorations and/or participatory experiments by executing their models within a web-based environment. The first tool on which an introductory tutorial is given is called The Model Exploration Module [1] or MEME, a generic tool that enables orchestrating experiments, managing results. MEME supports model analysis over a range of simulation platforms (RepastJ, NetLogo, Mason). It was designed to run large-scale parameter space explorations on grid/cloud systems or sensitivity analysis through statistical methods based on techniques known in the literature as Design of Experiments. The second framework is called the The Participatory Extension v2.0 [2] or PET v2.0, which is a further developed version of the original PET [3], a robust and generic web framework that allows modellers to extend their models to participatory simulations. It is a web application that incorporates agent-based simulations into a web interface compatible with any of the major web browsers, enabling users to administrate, run and participate in simulations in a way that they are familiar with, applying the mechanisms and practices they use every day while browsing web-pages and using other web-based applications. Applications of PET v2.0 may include online case studies for demonstrative and teaching purposes, or to conduct laboratory experiments for behavioural studies of a model. Prerequisites: The tutorial has no particular requirements, but experience in implementing agent-based models is an advantage. The frameworks are introduced through a supplied model, no programming is necessary. [1] Márton Iványi, László Gulyás, Rajmund Bocsi, Gábor Szemes, and Róbert Mészáros: "Model exploration module", In Agent 2007: Complex Interaction and Social Emergence Conference, Evanston, IL, USA, November 2007 [2] Richard Oliver Legendi, László Gulyás, Tamás Máhr, Rajmund Bocsi, Vilmos Kozma, Gábor Ferschl, Peter Rieger and Jakob Grazzini: "A New Set of Tools Supporting Agent-Based Economic Modelling". (Under publication, submitted to the 16th Portuguese Conference on Artificial Intelligence EPIA 2013, SSM – Social Simulation and Modelling Thematic Track) [3] Ivanyi, Marton, Rajmund Bocsi, Laszlo Gulyas, Vilmos Kozma and Richard Legendi. "The multi-agent simulation suite." In Emergent Agents and Socialities: Social and Organizational Aspects of Intelligence. Papers from the 2007 AAAI Fall Symposium, pp. 57-64. 2007.

1. Discussion - Matthias Meyer
Statistical and Behavioural Model Analysis Tools
Tamás Máhr, Richard Legéndi, László Gulyás
AITIA International, Inc.
tmahr@aitia.ai
rlegendi@aitia.ai
lgulyas@aitia.ai
Warsaw, 16th September 2013

2. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
A short introduction to designing and executing
computational experiments (i.e., simulations)
Show how to carry out DoE in practice (MEME)
Introduce the Participatory Extension (PET)
Aims of this talk
What are the practical challenges of designing and executing
computational experiments?
How to implement and execute the DoE approach in practice?
How to enable human subjects to participate in simulations?

3. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Dowload tools
http://bit.ly/essa2013

4. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Overview ABM research process

5. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Overview ABM research process
5

6. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Model Exploration
(Lorscheid – Heine – Meyer)

7. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
The General Approach
Computer simulations are experiments
Where the experimenter tries to determine
How the systems response (output) depends
On controllable factors (parameters)
One may also want to do replicates
(cf. RNG seeds)
System
(p1, p2, p3, p4, …) (r1, r2, r3, r4, …)

8. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Model Exploration
(Lorscheid – Heine – Meyer)

9. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Model Exploration
(Lorscheid – Heine – Meyer)

10. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Designing a simulation experiment (4)
Select a Factorial Design

11. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Designing a simulation experiment (4)
Select a Factorial Design
Select a design (or fill in the Design Table)
We will see a few further desings

12. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Practical Steps of (6)
Performing a Simulation Experiment
Set the parameters (factor values)
Combinations or levels
WHAT to record
Variables (time series)
Agent variables (changing length!)
Derived values (statistics)
WHEN to record
@end, @timestep, @N timesteps, @condition
WHEN to STOP the simulation
Fixed number of steps, condition reached, etc.
WHERE to execute
Local computer, local cluster, grid, cloud (comfort, pricing)

13. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Practical Steps of (6)
Performing a Simulation Experiment (cont'd)
Collect the results
When using more than a single core/computer, result files end up
dispersed
Assemble the result set
The ordering of the records (table rows) could be arbitrary
The number of columns may vary in raw output
(e.g., when recording raw agent variables)
Often one also needs to pre-process the result set
Aggregating, Splitting / Slicing
Archive the experiment
Keep a 'logbook' of your experiments
What results came from what experiment,
when and with what settings

14. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Model Exploration
How to implement and execute computational experiments?
Practicalities
Advanced Designs (beyond factorials)
Composite Central
Box-Behnken
Latin HyperCubes
„IntelliSweep“ tools
Iterative methods
Self-guided searches

15. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Full Factorial Designs
Classic parameter sweeps „as we know them”

16. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Full Factorial Design
A design in which every setting of every factor appears
with every setting of every other factor
A specialized version of the “brute force” strategy
Determines the same number of values (“levels”) for each
parameter (factor)

17. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Fractional Factorial Designs
Full factorial designs may be demanding even with two
levels only (k=10, 2k
=1024)
A fractional factorial experiment is
in which only an adequately chosen fraction of the parameter
combinations required for the complete factorial experiment
is selected to be run
Typically, we pick ¼, ½ of the full factorial

18. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Example: 2-Level Fractional Factorial Experiments with
MEME

19. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Example: 2-Level Fractional Factorial Experiments with
MEME

20. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Central Composite Design 1
The linear fit provided by the 2-level factorial methods
may not be enough
To build quadratic, or other higher-order models
we need new designs

21. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Central Composite Design 2
A factorial design
with added ‘star points’ on the axis of the parameters +
a center point
The star points can make new extreme values
for the parameters (both min and max)
The newly added points help to estimate the curvature

22. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Central Composite Design 3
There are three different types of CCDs:
Circumscribed (CCC)
Face centered (CCF)
Inscribed(CCI)
CCC and CCI are rotatable designs, because
every design point is at equal distance from
the center
The variance of the predicted response of a
model based on a rotatable design depends
only on the distance from the center point

23. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás

24. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
An Alternative Choice to Fit Quadratic Responses
The Box-Behnken design
an independent quadratic design
Does not contain an embedded
factorial or fractional factorial
Treatment combinations are
at the midpoints of edges
and at the center
A sphere
that protrudes through each face

25. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Properties of The Box-Behnken Design
Rotatable (or near rotatable)
Requires 3 levels of each factor
Have limited capability for orthogonal blocking
compared to the central composite designs

26. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Goals and Details of the Box-Behnken Design
The design should be sufficient to fit a quadratic model
The ratio of the number of experimental points to the number
of coefficients in the quadratic model should be reasonable
In fact, their designs kept it in the range of 1.5 to 2.6
The estimation variance
Should more or less depend only on the distance from the
centre
This is achieved exactly for the designs with 4 and 7 factors
Should not vary too much inside the smallest (hyper)cube
containing the experimental points

27. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Latin Hypercube Designs, 1
A screening method that
Easily handles more than 2 levels and
uses much less runs than the factorial design
LHD designs operate on
A subset of the parameter space defined by a single
contiguous interval for each dimension (parameter) –
a hypercube
The subset is defined by giving the low and high values for
each tested factor (parameter)

28. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Latin Hypercube Designs, 2
A criterion: non-collapsing design
If one of the parameters has (almost) no influence, then
two experiments that differ only in this parameter ‘collapse’
They are like measuring the same point twice
This is a waste of the resources (in deterministic cases)
Therefore, two design points should not share any
coordinate values
If it is not known a priori, which dimensions are important

29. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Latin Hypercube Designs, 3
Definition:
A d-dimensional grid of n levels in every dimension
Each level occurs only once
A non-collapsing design

30. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Latin Hypercube Designs, 4
A desirable property: Space filling
When no details on the functional behavior of the response
parameters are available, it is important to obtain information
from the entire design space
The points of the design should be ‘evenly spread’ over the
entire hypercube

31. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Latin Hypercube Designs, 5
A MAXIMIN design is a set of points, such that
The separation distance is maximal
I.e., the minimal distance among pairs of points
Assuming that the samples represent their ‘surroundings’ one
wants to make sure that we use our sample points efficiently
We maximize the r common radius of spheres around the
design points so that they don’t intersect
Any distance metric can be used, but L2 (Euclidean) is a
common choice

32. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
A Note on MAXIMIN LHDs
Finding a MAXIMIN, non-collapsing design
for many dimensions and
a high number of levels
is very hard
Therefore, often
pre-calculated designs are used, and/or
the MAXIMIN property is only approximated

33. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Latin Hypercube Designs in MEME
The LHD plugin in MEME supports
Up to 100 levels and
Up to 10 dimensions
Uses predefined experiment designs
Calculated by heuristic methods to approximate
MAXIMIN LHD designs
http://www.spacefillingdesigns.nl/maximin/info.html

34. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Latin Hypercube Designs in MEME

35. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Latin Hypercube Designs in MEME

36. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Dynamic “IntelliSweep” methods
So far, the entire design was fixed before starting the
experiment
There was no feedback from the measured responses to the design
Various (optimization) methods exist that use a different
strategy
Hill climbing, simulated annealing, genetic algorithms, etc.

37. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Iterative Uniform Interpolation 1
IUI is a response analysis method
Refines the parameter domain between iterations to achieve
better interpolation (of the response value)
Examines “interesting” subintervals by dividing them
further
Deviation from the previously observed (assumed) gradient
spans new measurements.

38. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Iterative Uniform Interpolation 2

39. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Genetic Algorithm Driven Methods
Optimization
Genetic algorithm (GA)
is a heuristic optimization method
F( o1
, …, on
) → max
Can be directly used for response analysis
If we are not interested in the entire response surface,
but only in high/low response values

40. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
MEME
The Model Exploration Module

41. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
MEME – The Model Exploration Module
Part of the Multi-Agent Simulation Suite (MASS)
Yet, useful with most major agent-based platforms (Repast J, Repast
Sym*, NetLogo, MASON, EMIL-S, FABLES)
GOAL:
User firendly toolset for ABM
Hiding coding / implementational difficulties as much as possible
Ease of use for non-technical people
MEME is responsible for
Design
Execution
Data collection
of computational experiments (i.e., agent-based simulations)

42. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
MEME – Functions
Assists the research process
from the point when the implementation of the model (or a version of
it) is complete
until the publication of the collected results
Helps configuring the simulation to record the proper variables
Data series from program variables or specific statistics of them
Offers wizards for a variety of experimental designs
Including fractional factorials and more
Orchestrates the execution of the experiment
on a single computer or on cluster or in the cloud
Collects the recorded data in standard data tables

43. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
MEME – Functions (cont'd)
Functions to preview the results
perform exploratory analysis,
preliminary charting
Export options and interfaces
To standard popular statistical packages like R, SPS, STATA
Optionally, a personal 'laboratory logbook'
Archiving and documenting the computational experiments
performed by the modeler

44. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
MEME History (since 2005)
2005 Tool to administer and process simulation results (Repast J & FABLES)
2006 Setting up config files for parameter sweeps (Repast J)
2007 Distributed execution on local clusters and grids (QosCosGrid)
2007 Design of Experiment (Classic Designs)
2008 Multi-Platform Support (EMIL-S, NetLogo, Custom Java, Repast Sym*)
2008 Advanced statistics in recording
2009 Standard Interface for results processing
2009 Advanced DoE designs (freely extensible architecture)
2010 Intellisweep plugins (iterative, self-guiding exploration of param space)
2011 Execution „in the cloud” (http://modelexploration.aitia.ai/)
2012 Support for the MASON simulation package
2013 MEME goes open source

45. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Design of Experiments in MEME
Classic simulation experiments with „parameter files“
Classic DoE tables
DoE Wizards
Factorials
Fractional Factorials
More
IntelliSweep experiments
Iterative methods
Self-guided searching of the parameter space
Optimization
E.g., Genetic Algorithms

46. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Simulation in the cloud
model exploration service
http://modelexplortaion.aitia.ai
runs RepastJ, NetLogo, Java models (Mason support is comming
soon)
uses Amazon EC2
user selects the number of machines used

47. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
MEME as a Personal Laboratory Logbook
The more one gets immersed in (computational) experiments,
the more results are filling the hard drive
„Nothing can be as alien/unknown than your own code –
after two months”
Same applies to experimental results and settings, let alone charts
„Hey, this is a nice chart. I wish I remembered what exact parameter
settings I user to run the model with!”
Being disciplined always helps, but tools may help being disciplined.
MEME stores all result sets in a DB (together with settings)
Grouped by model, version and „batch” (enforced)
You can add comments, remarks, descriptions to them

48. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Param Sweeps Param Sweeps Param Sweeps Param Sweeps
Results DB
Charts
Versioning and Merging
Filtering, Processing, Restructuring
Views
Export
(Excell, SPSS, etc.)
Import
(txt, csv, Excell, etc.)

49. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Practical
Start MEME
Run Simulation...
Open the El Farol model
Select experimental design
Define output
Run experiment
Analyze results

50. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
The El Farol bar problem
Parameters:
number of agents
overcrowding treshold
memory size
number of strategies
Agents are researchers (N=100)
They visit a popular but small bar in Santa Fé
If attendance > 60   (overcrowded)
Who hasn’t come  
If attendance <= 60  
Who hasn’t come  
Each day agents decide individually and in the same time

51. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
The El Farol model
MASON implementation of a NetLogo variant
Artifical agents:
ARMA-based prediction with history
Players have two actions: No go / go
If   +1 Score!
Goal: get max score

52. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Summary
Discussed the technical issues and challenges of executing
computational experiments
Explained the challenges of applying the DoE approach in practice
Introduced MEME as a tool to assist from the point when the
implementation of the simulation is complete
Discussed the usage of MEME
Together with advanced designs for computational experiments

53. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
PET
The Participatory Extension

54. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
About PET
Participatory Extension (PET)
Another component in the MASS toolbox
http://pet.aitia.ai/
Converts ABM's to
Web Simulations
Participatory Experiments
Laboratory experiments with human subjects
Some agents in the simulation are artificial, some others are
controlled by human agents

55. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Let's play!
http://demo1.aitia.ai

56. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Model List
Create
Experiments
Additional Info

57. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Subject list
(joined)
Admin tools
Admin page

58. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
In-experiment admin page
Player status
(moved/waiting)

59. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
In-experiment subject page
Status messages

60. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Post-experiment scores

61. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
PET software architecture

62. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
PET features
Runs Mason simulations
Can be scaled to run on multiple machines
Can be used:
Locally
Laboratory experiments
Policy makers (scenario analysis with a proper model)
On any webserver to run constantly
Gather data (scores from model and all user actions →
replay)
Dissemination
Questionnaire module
Verify if subjects understood the rules
Software is already in use by Universiteit van Amsterdam

63. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
PET requirements
An ABM model
PET is a generic framework
currently Mason models are supported

64. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
PET requirements
An ABM model
PET is a generic framework
currently Mason models are supported
Development of a web interface
no restricitions on tools (HTML5, Javascript, GWT, …)
communication is based on standards (AJAX)
config file maps incoming messages to method calls
GWT module available

65. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
PET communication
simple messages
invoked by user events (button click) and at initialization
getters, setters, etc.
call model methods
call agent methods
action messages
simulation is stopped when an action method (decision point!)
is hit
user events can send action messages
public void updateAttendance()
public void updateAttendance(final boolean attend)

66. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
PET communication
triggered messages
from simulation to browser
call model or agent method
configurable triggers
userActed (actionId)
messageHandled (messageId)
newTurn (actionId)
simulationEnd
userConnected (playerId, agentType)
userDisconnected (playerId, agentType)
unhandledExceptionCaught (exceptionType)

67. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
PET communication summary
broswer – simulation bidirectional communication
change / query the state of the model or your agent (simple
message)
make the simulation move on (action message)
have the simulation send data asynchronously (triggered
messages)

68. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Practical
download and start Eclipse
install the right GWT plugin from marketplace!
import the project you downloaded
open the GUI class

69. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Links to Software
http://meme.aitia.ai/
http://modelexploration.aitia.ai/
http://mass.aitia.ai
http://pet.aitia.ai/

70. Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás
Thank you!
Questions?
{tmahr|rlegendi|lgulyas}@aitia.ai