MEME – An Integrated Tool For Advanced Computational Experiments   Rajmund Bocsi 1 , Gabor Ferschl 1 , László Gulyás 1,2 ,...
Overview <ul><li>Agent-Based Modeling and  Computer Simulations </li></ul><ul><li>Exploration of Model Response </li></ul>...
Introduction
Agent-Based Modeling (ABM) <ul><li>Aims at creating complex (social) behavior  “from the bottom up” . </li></ul><ul><ul><l...
Validation and Verification <ul><li>Validation </li></ul><ul><ul><li>To do the right thing. </li></ul></ul><ul><li>Verific...
So, how to do it right? <ul><li>…  Methodology! </li></ul><ul><li>However, methodology is often constrained by the availab...
Why Tools? <ul><li>One can grind her own lens and  build her own microscopes… </li></ul><ul><li>... but standardized tools...
The General Approach <ul><li>Computer simulations are  experiments </li></ul><ul><ul><li>Where the experimenter tries to d...
Exploration of the Model Response (Parameter Sweeps, Batch Runs) <ul><li>Parameter-Response Analysis </li></ul><ul><li>Sto...
Problems
The Problem of Parameter Sweeps <ul><li>The required number of runs may be extremely large. </li></ul><ul><li>Reaching int...
The Simplest Solution to the High Computational Demand: Distribution <ul><li>Simulation runs with different parameter comb...
Advanced Distributed Solutions <ul><li>Distribution can be done on clusters  (or multiple cores in the same CPU) </li></ul...
A single cluster
A Utility Grid
An Example: the QosCosGrid <ul><li>Quasi-Opportunistic Supercomputing for Complex Systems Simulations on the Grid </li></ul>
A Desktop Grid Grid Server
An Alternative / Additional Solution
An Alternative / Additional Solution <ul><li>So far, we adopted a “brute force” approach </li></ul><ul><ul><li>Even in the...
The Challenge <ul><li>One cannot observe/measure the behavior of the system at all points in the parameter space </li></ul...
An Approach Smarter than  “Brute Force” <ul><li>In many disciplines, experiments may be very costly or hard (e.g., living ...
Tools:  The Model Exploration Module (MEME)
The Model Exploration Module <ul><li>Results Collection, Maintenance and Analysis </li></ul><ul><ul><li>Filtering, Variabl...
Param Sweeps Param Sweeps Param Sweeps Param Sweeps Results DB Charts Versioning and Merging Filtering, Processing, Restru...
Steps of Experiments with MEME <ul><li>Import results & analyze data  </li></ul><ul><ul><li>Create view tables and Charts ...
MEME Functions, Part 1 <ul><li>The user friendly wizard for </li></ul><ul><ul><li>Manual  setting up the parameter tree </...
MEME Functions, Part 2 <ul><li>The user friendly wizard for </li></ul><ul><ul><li>Automatic   setting up the parameter tre...
 
Example: 2-Level Fractional Factorial Experiments with MEME
Example: 2-Level Fractional Factorial Experiments with MEME
Example: Latin Hypercube Experiments with MEME
Example: Latin Hypercube Experiments with MEME
Future Works:  “IntelliSweep” Methods
Dynamic “IntelliSweep”methods <ul><li>So far, the entire design was fixed before starting the experiment </li></ul><ul><ul...
Iterative Uniform Interpolation 1 <ul><li>IUI is a response analysis method </li></ul><ul><ul><li>Refines the parameter do...
Iterative Uniform Interpolation 2
Genetic Algorithm Driven Methods O ptimization <ul><li>Genetic algorithm (GA)  is a heuristic optimization method </li></u...
Genetic Algorithm Driven Methods Active Non-linear Tests,  1 <ul><li>Active Non-linear Tests (ANTs) were proposed by  John...
Genetic Algorithm Driven Methods Active Non-linear Tests,  2 <ul><li>Measure “fit” of the sample to the thesis </li></ul><...
Multi-Platform Support <ul><li>MEME imports results from virtually any format </li></ul><ul><li>MEME can master computatio...
Summary <ul><li>Correct methodology is vital for computational experiments </li></ul><ul><li>Methodology is often dependen...
Acknowledgements <ul><li>This work benefited from the partial support of the following grants </li></ul><ul><ul><li>The EL...
Availability of Tools <ul><li>http://mass.aitia.ai/ </li></ul><ul><ul><ul><li>http://fables.aitia.ai/ </li></ul></ul></ul>...
Upcoming SlideShare
Loading in …5
×

MEME – An Integrated Tool For Advanced Computational Experiments

1,067 views
902 views

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,067
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

MEME – An Integrated Tool For Advanced Computational Experiments

  1. 1. MEME – An Integrated Tool For Advanced Computational Experiments Rajmund Bocsi 1 , Gabor Ferschl 1 , László Gulyás 1,2 , Attila Szabó 1,2 1 AITIA International Inc. 2 Lorand Eotvos University, Budapest
  2. 2. Overview <ul><li>Agent-Based Modeling and Computer Simulations </li></ul><ul><li>Exploration of Model Response </li></ul><ul><ul><li>Problems and Solutions </li></ul></ul><ul><li>The Model Exploration Module (MEME) </li></ul><ul><li>Future Works </li></ul><ul><ul><li>Dynamic Methods </li></ul></ul><ul><li>Summary </li></ul>
  3. 3. Introduction
  4. 4. Agent-Based Modeling (ABM) <ul><li>Aims at creating complex (social) behavior “from the bottom up” . </li></ul><ul><ul><li>Complex interactions of a high number of </li></ul></ul><ul><ul><li>(complex) individuals . </li></ul></ul><ul><ul><li>A generative and mostly theoretical approach: </li></ul></ul><ul><ul><ul><li>Computational “thought experiments” , </li></ul></ul></ul><ul><ul><ul><li>Existence proofs, etc. </li></ul></ul></ul>
  5. 5. Validation and Verification <ul><li>Validation </li></ul><ul><ul><li>To do the right thing. </li></ul></ul><ul><li>Verification </li></ul><ul><ul><li>To do it right. </li></ul></ul>
  6. 6. So, how to do it right? <ul><li>… Methodology! </li></ul><ul><li>However, methodology is often constrained by the available / used tools . </li></ul>
  7. 7. Why Tools? <ul><li>One can grind her own lens and build her own microscopes… </li></ul><ul><li>... but standardized tools make routine tasks much easier and safer (easier to be correct). </li></ul><ul><ul><li>Moreover, validation is easier. (No need to report on the lens grinding process, methods, etc.) </li></ul></ul><ul><li>Also applies to very general tools </li></ul><ul><ul><li>C, Java, etc. </li></ul></ul><ul><ul><li>Mathematica </li></ul></ul>
  8. 8. The General Approach <ul><li>Computer simulations are experiments </li></ul><ul><ul><li>Where the experimenter tries to determine </li></ul></ul><ul><ul><li>How the systems response (output) depends </li></ul></ul><ul><ul><li>On controllable factors (parameters) </li></ul></ul><ul><ul><li>One may also want to do replicates (cf. RNG seeds) </li></ul></ul>System (p 1 , p 2 , p 3 , p 4 , …) (r 1 , r 2 , r 3 , r 4 , …)
  9. 9. Exploration of the Model Response (Parameter Sweeps, Batch Runs) <ul><li>Parameter-Response Analysis </li></ul><ul><li>Stochastic components </li></ul><ul><ul><li>Very common in modeling: concepts not modeled explicitly </li></ul></ul><ul><ul><li>In our example: </li></ul></ul><ul><ul><ul><li>Initialization </li></ul></ul></ul><ul><ul><ul><li>Execution order </li></ul></ul></ul><ul><li>Sensitivity Analysis </li></ul><ul><ul><li>Dependence on parameter values </li></ul></ul>
  10. 10. Problems
  11. 11. The Problem of Parameter Sweeps <ul><li>The required number of runs may be extremely large. </li></ul><ul><li>Reaching into the tens or hundreds of thousands. </li></ul>
  12. 12. The Simplest Solution to the High Computational Demand: Distribution <ul><li>Simulation runs with different parameter combinations can be run </li></ul><ul><ul><li>In parallel, </li></ul></ul><ul><ul><li>On different computers. </li></ul></ul><ul><li>Due to the independence of runs. </li></ul>
  13. 13. Advanced Distributed Solutions <ul><li>Distribution can be done on clusters (or multiple cores in the same CPU) </li></ul><ul><ul><li>But one needs to own all the computers for that. </li></ul></ul><ul><li>Or, distribution can be done on “grids” </li></ul><ul><ul><li>Grids are “cluster of clusters” </li></ul></ul><ul><ul><ul><li>A loosely assembled pool of computers in different “administration domains” </li></ul></ul></ul><ul><ul><ul><li>Utility versus Desktop grids </li></ul></ul></ul>
  14. 14. A single cluster
  15. 15. A Utility Grid
  16. 16. An Example: the QosCosGrid <ul><li>Quasi-Opportunistic Supercomputing for Complex Systems Simulations on the Grid </li></ul>
  17. 17. A Desktop Grid Grid Server
  18. 18. An Alternative / Additional Solution
  19. 19. An Alternative / Additional Solution <ul><li>So far, we adopted a “brute force” approach </li></ul><ul><ul><li>Even in the most advanced distributed case. </li></ul></ul><ul><li>We explored all combinations </li></ul><ul><ul><li>Of all values for </li></ul></ul><ul><ul><li>All parameters (parameter space dimensions) </li></ul></ul>
  20. 20. The Challenge <ul><li>One cannot observe/measure the behavior of the system at all points in the parameter space </li></ul><ul><ul><li>We can only take samples . </li></ul></ul><ul><li>In most practical cases, we can’t even explore all possible/meaningful/relevant values of single parameters. </li></ul>
  21. 21. An Approach Smarter than “Brute Force” <ul><li>In many disciplines, experiments may be very costly or hard (e.g., living animals) </li></ul><ul><li>Design of Experiments (DoE) introduced by R. Fisher (1920s) </li></ul><ul><ul><li>methods that reduce the number of experiments and still yield statistically significant results. </li></ul></ul>
  22. 22. Tools: The Model Exploration Module (MEME)
  23. 23. The Model Exploration Module <ul><li>Results Collection, Maintenance and Analysis </li></ul><ul><ul><li>Filtering, Variable Selection, Aggregation and Statistics </li></ul></ul><ul><ul><li>Scripting for advanced users </li></ul></ul><ul><ul><li>Numerous charting and exporting options </li></ul></ul><ul><li>Parameter Space Explorations Intelligently and Efficiently </li></ul><ul><ul><li>On a single computer </li></ul></ul><ul><ul><li>On local clusters </li></ul></ul><ul><ul><li>On grid middleware </li></ul></ul><ul><li>Design of Experiment (DoE) Support </li></ul><ul><ul><li>Period2 activities </li></ul></ul>
  24. 24. Param Sweeps Param Sweeps Param Sweeps Param Sweeps Results DB Charts Versioning and Merging Filtering, Processing, Restructuring Views Export (Excell, SPSS, etc.) Import (txt, csv, Excell, etc.)
  25. 25. Steps of Experiments with MEME <ul><li>Import results & analyze data </li></ul><ul><ul><li>Create view tables and Charts </li></ul></ul><ul><ul><li>(Potentially automatically or ‘batched’) </li></ul></ul><ul><li>Potentially export the results of your analysis </li></ul><ul><ul><li>(View) Tables or Charts </li></ul></ul><ul><li>Load the model (automatic) </li></ul><ul><li>Set the parameter tree </li></ul><ul><ul><li>manually, </li></ul></ul><ul><ul><li>(or by configuring automated sweep methods) </li></ul></ul><ul><li>Set recordable values and statistics </li></ul><ul><li>Run the experiment </li></ul>
  26. 26. MEME Functions, Part 1 <ul><li>The user friendly wizard for </li></ul><ul><ul><li>Manual setting up the parameter tree </li></ul></ul><ul><ul><li>Specifying recordable elements </li></ul></ul><ul><li>Simulation execution </li></ul><ul><ul><li>On single computers </li></ul></ul><ul><ul><li>Local clusters </li></ul></ul><ul><ul><li>And possibly on grids </li></ul></ul><ul><li>Result analysis </li></ul>
  27. 27. MEME Functions, Part 2 <ul><li>The user friendly wizard for </li></ul><ul><ul><li>Automatic setting up the parameter tree </li></ul></ul><ul><ul><ul><li>Design of Experiments plugins </li></ul></ul></ul><ul><ul><ul><li>Dynamic IntelliSweep methods </li></ul></ul></ul><ul><ul><li>Specifying recordable elements </li></ul></ul><ul><ul><ul><li>Together with advanced statistics and on-the-fly code generation </li></ul></ul></ul><ul><li>Simulation execution </li></ul><ul><li>Result analysis </li></ul><ul><ul><li>Advanced export options </li></ul></ul>
  28. 29. Example: 2-Level Fractional Factorial Experiments with MEME
  29. 30. Example: 2-Level Fractional Factorial Experiments with MEME
  30. 31. Example: Latin Hypercube Experiments with MEME
  31. 32. Example: Latin Hypercube Experiments with MEME
  32. 33. Future Works: “IntelliSweep” Methods
  33. 34. Dynamic “IntelliSweep”methods <ul><li>So far, the entire design was fixed before starting the experiment </li></ul><ul><ul><li>There was no feedback from the measured responses to the design </li></ul></ul><ul><li>Various (optimization) methods exist that use a different strategy </li></ul><ul><ul><li>Hill climbing, simulated annealing, genetic algorithms, etc. </li></ul></ul>
  34. 35. Iterative Uniform Interpolation 1 <ul><li>IUI is a response analysis method </li></ul><ul><ul><li>Refines the parameter domain between iterations to achieve better interpolation (of the response value) </li></ul></ul><ul><li>Examines “interesting” subintervals by dividing them further </li></ul><ul><ul><li>Deviation from the previously observed (assumed) gradient spans new measurements. </li></ul></ul>
  35. 36. Iterative Uniform Interpolation 2
  36. 37. Genetic Algorithm Driven Methods O ptimization <ul><li>Genetic algorithm (GA) is a heuristic optimization method </li></ul><ul><ul><li>F( o 1 , …, o n ) -> max </li></ul></ul><ul><li>Can be directly used for response analysis </li></ul><ul><ul><li>If we are not interested in the entire response surface, but only in high response values. </li></ul></ul>
  37. 38. Genetic Algorithm Driven Methods Active Non-linear Tests, 1 <ul><li>Active Non-linear Tests (ANTs) were proposed by John H. Miller of CMU and SFI </li></ul><ul><li>A response analysis method that uses GA to disprove (prove) a user-defined thesis </li></ul><ul><ul><li>Thesis: P -> {(o 1 , …, o n ) | <condition>} </li></ul></ul><ul><ul><li>Sample by GA: P -> {(o 1 , …, o n ) | (p 1 , …, p m )  P m } </li></ul></ul>
  38. 39. Genetic Algorithm Driven Methods Active Non-linear Tests, 2 <ul><li>Measure “fit” of the sample to the thesis </li></ul><ul><ul><li>Breed parameter combinations that are farthest from the thesis </li></ul></ul><ul><li>In the end, the farthest sample provides a level of falsification </li></ul><ul><ul><li>Notice that an effective non-linear optimization method is being used to falsify the thesis! </li></ul></ul>
  39. 40. Multi-Platform Support <ul><li>MEME imports results from virtually any format </li></ul><ul><li>MEME can master computational experiments of simulations written in </li></ul><ul><ul><li>Repast J </li></ul></ul><ul><ul><li>FABLES </li></ul></ul><ul><ul><li>NetLogo </li></ul></ul><ul><ul><li>Custom Java </li></ul></ul><ul><ul><li>Repast Simphony* </li></ul></ul>
  40. 41. Summary <ul><li>Correct methodology is vital for computational experiments </li></ul><ul><li>Methodology is often dependent on the availability of tools </li></ul><ul><li>The Model Exploration Module is a free tool to support computational modelers </li></ul><ul><ul><li>In methodologically exploring the response of their computational models </li></ul></ul>
  41. 42. Acknowledgements <ul><li>This work benefited from the partial support of the following grants </li></ul><ul><ul><li>The ELTE Informatics Cooperative Research and Education Center (GVOP-3.2.2-2004.07-005/3.0) by the Hungarian Government; </li></ul></ul><ul><ul><li>The FP6 STREP project QosCosGrid (contract #033883) by the European Commission; </li></ul></ul><ul><ul><li>The FP6 STREP project Emergence in the Loop (EMIL) (contract #033841) by the European Commission. </li></ul></ul>
  42. 43. Availability of Tools <ul><li>http://mass.aitia.ai/ </li></ul><ul><ul><ul><li>http://fables.aitia.ai/ </li></ul></ul></ul><ul><ul><ul><li>http://meme.aitia.ai/ </li></ul></ul></ul><ul><li>Downloadable free of charge </li></ul><ul><ul><li>With full documentation </li></ul></ul>

×