ICPSR - Complex Systems Computing - Session 9 - 2011

Analyzing Agent-‐Based Models: Part I Charles Doriean Department of Poli<cal Science Center for the Study of Complex Systems University of Michigan Introduc<on to Compu<ng for Complex Systems (Session IX)

Overview •  What You’ve Learned So Far This Week •  Where Do We Go From Here? –  Approaches to exploring the parameter space (and building a model with the parameter space in mind) –  Iden<fying model outputs to measure •  Tomorrow: What Do I Do With All This Data? –  Iden<fying model outcomes, and robustness/ sensi<vity of results

What You’ve Learned This Week (about ABMs) •  What they are, and what they can be used for •  The building blocks in NetLogo –  Turtles (agents) and what they can do –  The grid (if needed) and its characteris<cs –  How to make agents interact, and connect them to a grid (again, if needed) –  Visualiza<ons of model processes (graphs, grid displays, etc.)

What You’ve Learned This Week (about Model Analysis) •  How to manipulate NetLogo sliders to do a “quick and dirty” sweep of a single parameter •  How to use NetLogo packages to set up a sweep of a parameter space (e.g. many itera<ons at each permuta<on of values on mul<ple parameters) •  How to get the data into a spreadsheet –  Pre[y easy with CSV format, but may need some cleanup, depending on how you format output

A Li[le Terminology •  Parameters: Quan<<es you set exogenously •  Variables: Quan<<es that change with model play (i.e. endogenous) •  Assump<ons: how parameters aﬀect variables, and how variables aﬀect other variables •  Experiment: The systemic inves<ga<on of model results over a parameter space

On Parameter Spaces •  An experiment is designed to inves<gate the nature of model results over a parameter space •  They expand very quickly –  Addi<onal parameters and addi<onal values cause the space to expand geometrically •  Job sizes grow quickly, too –  Addi<onal runs at each parameter combina<on cause the length of the job to expand linearly, as does increasing the length of each run •  Paralleliza<on helps, but you also want experiments to be tractable so that others can replicate them readily

Some Approaches to Making Parameter Spaces Tractable •  A few possibili<es: –  Reduce the number of values you use for each parameter –  Choose an exogenous sampling strategy for the parameter space –  Let your sampling strategy be endogenous to the results from previous runs –  (Philosophical) Remove parameters

Parameter Value Reduc<on •  Could choose wider intervals between values –  Careful: this approach assumes that you’re not missing anything important in the gaps •  Could choose to disregard some por<ons of the parameter space, in favor of more “interes<ng” ones –  Ques<on: what does it mean to have an “uninteres<ng” part of the space? –  Also, which parts are interes<ng?

Sampling the Parameter Space •  Could do this many ways: –  1. Random sampling •  Just pick parameter combina<ons at random, un<l you’ve ﬁlled out the outcome space reasonably –  2. La<n hypercube sampling •  Pick combina<ons so that each value of each parameter is sa<sﬁed at least once –  3. Orthogonal sampling •  Like 2, but also span subspaces (Sudoku) •  May not be so good with stochas<c models, where you want condi<onal distribu<ons of behavior at par<cular parameter values

Choose Parameter Values Endogenously •  Example: Ac<ve Nonlinear Tes<ng (Miller 1998) –  h[p://zia.hss.cmu.edu/miller/papers/ant.pdf •  Basically a meta-‐model that employs a search algorithm within the parameter space (or assump<on space) that tries to “break” the model’s results

Parameter Space Reduc<on •  How many parameters does my model need? –  Can I eliminate one without losing much generality? •  Which ones are cri<cal to my research ques<on? •  Which ones are cri<cal to comparability with other models (especially exis<ng models?) –  Are some parameter combina<ons equivalent to special cases? (ohen the case with corner values, especially when a parameter is a probability) –  If I add several “new pieces” to my model, how will I know which ones are driving my results? Should maybe make only one departure at a <me (Abe Gong’s idea)

What Outputs Should I Collect? •  Answer: depends on why you built the model •  Two major mo<va<ons (though, there could be more:) –  Building intui<ons about how social processes work –  Genera<ng observable implica<ons (for tes<ng on exis<ng data or forecas<ng)

Building Intui<on •  ABMs are powerful tools for building understanding of social processes, and that understanding is useful in its own right •  Visual iden<ﬁca<on of qualita<ve pa[erns (via GUIs) can lead us to key in on quan<<es to inves<gate more thoroughly in an “experiment” •  Good reason to collect intermediate variables, even ones that aren’t observable in real-‐life (es<ma<ng unobservable parameters is a good reason to model, in general)

Genera<ng Observable Implica<ons •  Can also think of a model as a bridge between observable quan<<es (e.g. an exogenous parameter and an endogenous variable) that we think are linked in some way •  What is the nature of that link (e.g. Posi<ve? Nega<ve? How strong?) •  Even if some of your parameters aren’t directly related to your research ques<on, they can be useful as robustness checks for empirical claims

Characteris<cs of Model Output •  Does the model se[le down? If so, where in the outcome space does it se[le down? How long does it take to se[le down? –  Could be phrased probabilis<cally (e.g. where is it most likely to se[le down?) •  If it doesn’t se[le down, what does it do? –  Cycles with ﬁnite periods (e.g. repe<<on) –  Cycles with inﬁnite periods (chaos) –  Complexity (somewhere between order and chaos)

Characteris<cs of Model Output (cont’d) •  Ac<vity: Schelling’s segrega<on model and the sensi<vity threshold for changing loca<on –  Does the model “stop” at a given value for threshold? –  How could you tell, without using the GUI (that is, what data would you collect, and how would you interpret it?) –  What about <me scales? How long must runs be to determine whether a model “stops” or not?

Characteris<cs of Model Output (cont’d) •  How do parameters interact to produce outcomes? –  We use nested parameter sweeps to inves<gate combina<ons of parameter values, and it’s not likely that model outcomes are simply a linear combina<on of each parameter’s eﬀect •  This is a thornier problem –  Can start to get at it with cross-‐tabs, condi<onal sta<s<cs, etc. –  But a more thorough sensi<vity analysis may be in order – more on this tomorrow

ICPSR - Complex Systems Computing - Session 9 - 2011

by Daniel Katz on Jul 28, 2011

Accessibility

Categories

Upload Details

Usage Rights

Statistics

ICPSR - Complex Systems Computing - Session 9 - 2011 — Presentation Transcript