Chapter7 abm book_noha_nagi

© Noha Nagi- March 2017
Chapter 7: Verification, Validation and Replication
Examining the correctness and utility of an Agent-Based Model
An introduction to Agent-based Modelling book

Essentially, all models are wrong, but some are useful.
— George Box
“
“

Examining the
correctness of
a model
• If a model is to be useful for answering real-world questions, it is important that the
model provides outputs that address the relevant issues and that the outputs are
accurate: The model must provide outputs that are useful to the model user.
• There are three ways to evaluate the model accuracy and utility:
Is the model free of bugs?
Can we replicate the model
with the same code?Does the model correspond to and
explain a phenomenon in the real world?
&correspond to the conceptual model?
If the model is verified and validated, the confidence grows in
the correctness and explanatory power of both the conceptual and implemented model
Note1: scientific community tend to accept the model as correct (without replication)
Note2: Statistical methods are needed to illuminate the stochastic behavior of the ABM (Multiple runs)

Model
Verification
• The process of verification is easier when the model designer &
implementer are the same person (ideal situation) and difficult otherwise.
Lately with Netlogo and the smaller time needed to learn it, the ideal situation is
possible.
Otherwise, and earlier in research, they were different persons. Communication
here is highly important. Additional type of error appears: Misunderstanding.
To decrease this type of error:
The domain expert needs to know the basics of
the programming environment to help in formulating the conceptual model.
The programmer need to know the conceptual model to help in implementing
the simplifications and avoid pitfalls in implementation
Goal:
eliminate the
coding bugs

Model
Verification
• The process of verification is easier when the model building is incremental.
Build incrementally from simple to complicated. Verify on blocks and additional blocks…
Does this guarantee the accuracy of the code?
How?

The Voting Model
Application on

Model
description• Social interaction determines the voting decisions in elections
• The time between the voting decision and the expression in actual voting is
spend on discussions. This/these discussion(s) can affect the voting decision.
Assuming a simple model with two voting options (0 or 1)

Model
description• It is important to document well the conceptual model such that
implementation is smooth and verification becomes an easier task.
• Different ways:
Flowchart
Pseudo-code
UMLGraphical and NL

Model
verification• One way to verify the model is to perform unit testing
That is we check if every single procedure (individual units) does exactly
what it is meant to do.
Example on the voting model:
setup
set 50% of voters to 0
Unit Testing

Model
verification
we verify if this brings equal number from
different voters
Calculate and find if it the difference >10 %
Check-setup is a procedure that is called in the SETUP procedure
Unit Testing

Model
verification
• Unit testing helps in re-writing a better code with fewer bugs.
• Unit testing ensures that future changes to our code, won’t distrust previous code.
Unit Testing

Model
verification
Unit Testing
go
Change the vote if more that 4 neighbors have a different vote
Discussion: How to verify this?

Model
verification
• There is also online /in-model unit testing
Not part of the model code
Used will model is running
Unit Testing
• There is also offline unit testing
Not part of the model code
Used to test with specific model inputs and compare to the expected results
Patton, 2005

Model
verificationA bug or an emergent property?
Hypothesis:
Static Blocks of Voters with Smooth edges
Result:
Jagged edges
Never reaches equilibrium – continually cycles through
a set of states
Beyond

Model
verification
Checking back the code.
It is a property.
The reason is: tie votes
Beyond
Block Edges will be always unstable
Turing back and forth between the two votes
If the CHANGE-VOTE-IF-TIED? switch is
on, then in the case of a tie, the central
patch will always change its vote.
Here, from 8 neighbors, if half of them are of a certain vote, the patch will be
hesitant
It will change its vote if the 4 are not only neighbors but are also ties
Here no network, it is a probability --- and it alters
Experts added this as a new
meaningful voting behavior

Model
verification
Checking back the code.
It is a property.
However, it is sometimes hard to know
Beyond
Code error?
Miscommunication between Expert & Model implementer?
Possible result?

Model
robustness
Goal: understanding how the model is sensitive/robust to various conditions
In the voting example, the interesting results were shown in the verification
process.
How does the model behave with changes in initial conditions?
This happens by:
• Varying the values of parameters that was already in the model
• Adding new parameters to the model
Note: Trials of SA automation (ANT methodology, ex)
… but BehaviorSpace is also beneficial
Sensitivity analysis

Model
robustness
Slider is added, to control the distribution of votes and check the effect of non-
balanced distribution of votes.
Varying the values of parameters that was already in the model

Model
robustness
Using behavior space, a number of experiments are done
Varying the percent from 25 to 75 % with 5% increment
Above 70-75%, there is dominance –try!
Stop the experiments when:
• No voter changed vote at the last time step
• After 100 time points
--- It may be never-ending/oscillating or it may reach a steady state
Side note: Alter the code to include a unit test with a parameter

Model
robustness
Dominance of green at >70%
Each point is an average of 10 runs (repetitions)
The model is sensitive to the initial distribution of voters Sensitivity analysis

Model
robustnessThe model is sensitive
The model is not sensitive
Quantitative result
Qualitative result
Solid islands of voters are formed

Model
robustness
Adding new parameters, agent mechanisms, environmental conditions to the model
What if other topology than a 2D grid is applied?
A network topology? A hex grid? Or other?

Model
verification
Benefits
• Developing an understanding for the cause of unexpected outcomes
• Exploring the impact of small changes in rules.
Benefits and issues
Precautions
• Be rigorous, that is the only way to perform a good verification process.
• It is the role of the researcher/model implementer to decide when is the
verification process is sufficient to provide a correspondence between the
model and the conceptual model.
• Admit the possible error in code. Unexpected results can be an error.
There is a continuum of verification and you will always be between.

Model
validation
Compare the implemented model with reality.
Only in the aspects that are important to your research questions.
Models are not reality. They are simplifications. There is a continuum of
validation.
The model is valid/invalid relative to the questions it’s used to answer.

Model
validation
Types
Based on validation level:
• Microvalidation: making sure the behaviors and mechanisms encoded into
the agents in the model match up with their real world analogs.
• Macrovalidation is the process of ensuring that the aggregate, emergent
properties of the model correspond to aggregate properties in the real world.
Based on details level:
• Face validation: process of showing that the mechanisms and properties of
the model look like mechanisms and properties of the real world.
• Empirical validation makes sure that the model generates data that can be
demonstrated to correspond to similar patterns of data in the real world
In my thesis:
Macro and
Empirical
validation

The Flocking Model
Application on

Model
descriptionAlignment
Separation
No leadership. Each bird follow a set of rules and flocking patterns emerges.
Cohesion
Same direction as
nearby birds
Avoid hitting
another bird
Move toward other
nearby birds
Are these rules
modelling real birds
behavior?

Model
Micro
Validation
Start from comparing the model actions and data at the lowest level
Examine all of the relevant properties of the birds, such as
location in the world, heading, flockmates, and nearest neighbor,
using NetLogo ’ s inspector
Real birds also have locations and headings and, based on
their vision, are aware of the birds around them.
However, real birds have age, health and can eat.
Real birds also move in three dimensions, but our bird
agents only move in two.
Research question: Flocking
Then include location, heading, vision
Does it worth to make it in 3D?
Examine

Model
Macro
Validation
Here investigating the relationship between the global properties of the model
and the flocking patterns of real birds,
Does the flocking pattern similar to real patterns?
Yes. They also seem to interact with each other in other
ways that we would expect.
We could also calculate the
average flock size in model
to see if it corresponds to the
average flock size in the
real world.

Between
Model
Micro & macro
Validation
Macrovalidation tells you if you have captured the important parts of the system,
whereas
Microvalidation tells you if you’ve captured the important parts of the agent’s
individual behavior
Power of ABM vs. traditional models
Various levels of validation
• showing that individual birds behave as expected
• ask if flockmates react to each other as expected
• determine the validity of the behavior of the flocks as a whole.

Model
Face
Validation
Ensures that someone who looks at the model “ on face ” (i.e.,without detailed
analysis) can easily be convinced that the model contains elements and components
that correspond to agents and mechanisms that exist in the real world.
In the flocking model, on the surface, the birds appear to behave like real birds:
They generally move in a straight line, have the ability to change directions, and
make decisions based on local information.
Appearance of microbehaviorface micro validity
Appearance of microbehavior face macro validity

Model
Empirical
Validation
Data produced by the model must correspond to empirical data derived from the
real-world phenomenon.
Involves running statistical tests on the comparable data sets to determine how
similar these data sets are.
Mean, SD, Student’s t-test and the Kolmogorov-Smirnov test
Comparing how quickly birds in the model can turn compared to real birds Empirical micro validity
Testing with real-world data on pattern  Empirical macro validity
Pattern-oriented modelling is a type of empirical validation that compares the pattern of model vs. real
data (graphically). A perfect match happens if all every point were matched.

Model
Calibration
The process of finding the parameters and initial conditions that cause the
model to match up with real-world data
It involves using empirical data to adjust the parameters and mechanisms of a model
so that it can be used to examine a particular situation.

Notes
• Agent-based models are usually stochastic in nature, rarely producing the same
result twice even given the same initial parameters.
• That’s why we use standard statistical tools such as the t -test and the
Kolmogorov-Smirnov test to determine if our model is producing a distribution
consistent with that produced in the real world.
• Some models are hard to validate with data. They have variant results.
Invariant results are those that occur no matter how many times we run the
model. Variant results, by contrast, are those that change depending upon how
the model evolves.
• Partition the space of model results into variant and invariant areas, this helps
you in validation.
• Agent-based models usually include path-dependent processes. This may cause
the variant results.

Model
Verification
vs.
validation
summary

Model
replication
• If a model can be implemented again, using the same conceptual model,
providing similar (not exactly the same) results &/or conclusion (i.e. successful
replication), then this will provide confidence on the original model: it’s
verification and validation.
• Replication is important in refining the original model (conceptual
model/assumptions/further explorations and conclusions…).
• Replication also provides researchers with a shared understanding of
modelling, its concepts, terminology....

Model
replication
• The implementation of the replicated model must differ in some way from the
original model and must also be executable. There is 6 ways –at least- for
variations:
Different
conclusions
mean
Language Toolkit Hardware
Algorithm Author
Original model is sensitive to small changes of
the order of behavior execution
It should be robust to
such changes…
Ex: Search Algorithm This is a strong test!
Time
Original model is
inadequate

Model
Replication
standard
• Replication standard (RS) serves as a criterion for replication success. Three
categories of RS:
1. Numerical identity: is difficult to establish, as it entails showing that the
original and replicated models produce the exact same numerical results. Seeds
can be used here.
2. Distributed equivalence: the goal is to show that the two implemented models
are sufficiently similar to each other according to statistical measures.
3. Relational alignment: exists if the results of the two implemented models show
qualitatively similar relationships between input and output variables. If x
increase, y responds in the same way for both models.

Recommendations
for model
replicators
Better
delayed!
May mean that the model
is not documented well

Recommendations
for model
authors
SA imp.
It’s better that experts build the models themselves. Netlogo helps. Or collaborate well as a team:
use Netlogo modeling commons.

Replication overwrites!

Chapter7 abm book_noha_nagi

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Chapter7 abm book_noha_nagi

Similar to Chapter7 abm book_noha_nagi (20)

Recently uploaded

Recently uploaded (20)

Chapter7 abm book_noha_nagi