A talk given at the CECAN workshop on "What Good Data could do for Evaluation" at the Alan Turing Institute, 25th Feb. 2019. Abstract: In complex situations (which includes most where humans are involved) it is infeasible to predict the impact of any particular policy (or even what is probable). Randomised Control Trials do not tell one: what kinds of situation a policy might work in, what are enablers and inhibitors of the effectiveness of a policy. Here I suggest that using 'fat' data and simulation might allow a possibilistic analysis of policy impact - namely an exploration of what could go surprisingly wrong (or indeed right). Whilst this does not allow the optimisation of policy, it does inform the effective monitoring of policy, and basic contingency planning. However, this requires a different approach to policy - from planning and optimisation to an adaptive approach, with richer continual monitoring and a readiness to tune or adapt policy as data comes in. Examples of this are given concerning domestic water consumption (in the main talk), and in supplementary slides: voter turnout and fishing.

1. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 1
Mixing fat data, simulation and policy
– what could possibly go wrong?
Bruce Edmonds
Centre for Policy Modelling
Manchester Metropolitan University

2. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 2
Motivation and Issues
Part 1:

3. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 3
Lessons from robotics: Part I
Robotics in the 70s and 80s tried to (iteratively):
1. build a map of its situation (i.e. a predictive
model)
2. use this model to plan its best action
3. then try to do this action
4. check it was doing OK go back to (1)
But this did not work in any realistic situation:
• It was far too slow to react to its world
• to make useable predictions it had to make too
many dodgy assumptions about its world

4. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 4
Lessons from robotics: Part II
Rodney Brooks (1991) Intelligence without
representation. Artificial Intelligence, 47:139–160
A different approach:
1. Sense the world in rich fast ways
2. React to it quickly
3. Use a variety of levels of reaction
a. low simple reactive strategies
b. switched by progressively higher ones
Do not try to predict the world, but react to it quickly
This worked much better.

5. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 5
Lessons from Weather Forecasting
• Taking measurements at a few places and trying
to predict what will happen based on simple
models based on averages does not work well
• Understanding the weather improved with very
detailed simulations fed by rich and
comprehensive sensing of the system
• Even then they recognize that there are more
than one possibilities concerning the outcomes
(using ensembles of specific outcomes)
• If these indicate a risk of severe weather they
issue a warning so mitigating measures can be
taken even if

6. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 6
Lessons from Radiation Levels
• The human body is a very complex system
• It has long been known that too much radiation
can cause severe illness or death in humans
• In the 30s & 40s it was assumed there was a
“safe” level of radiation
• However it was later discovered that any level of
radiation carried a risk of illness
• Including naturally occurring levels
• Although an increase in radiation might not seem
to affect many people, it did result in more
illnesses in some

7. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 7
Prediction in Complex Situations
• For many kinds of complex situation or issue
prediction (in the sense of correctly anticipating
what would happen) is impossible for most
desired indicators of outcomes
• There are many reasons for this, including: chaos,
structural change, multiple assumptions,
cascading social effects, context-sensitivity etc.
• In particular, this is true of many social systems,
where individuals influence each other
• This is impossible even probibilistically, that is
completely different outcomes may happen

8. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 8
Finding the best policy beforehand
• A lot of policy making (e.g. the classic process in
the Magenta book) relies on being able to
anticipate the costs and benefits of policy before
they are implemented, to inform the choice
• However, in cases where prediction of the impact
of policy is impossible, then this approach fails
• This includes any optimisation of policy
• This talk looks at some of the ways we might
approach policy evaluation and monitoring in such
situations and the role that data can have in this

9. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 9
Kinds of Data
Part 2:

10. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 10
Randomised Control Trials (RCT)
• Tells you whether any measurable indicator is
affected by particular interventions or policies with
relatively few assumptions
• Difficult to fudge due to wishful thinking
• Strong at showing a proposed intervention or
policy does not work as hoped at the macro level
• But it is specific to the set of circumstances it was
tried in – it does not tell one why the outcomes
happen or whether it will work in other situations
• It will not tell you about the enablers (what makes
the desired outcome possible) or the frustrators

11. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 11
Pure Qualitative Evaluation
• Gives rich accounts of what happened, why
people did things and their perceptions of
outcomes
• Useful with stakeholders for collecting a variety of
views and evidence which otherwise might be
missed
• Does not provide a representative sample
• Is vulnerable to many human & micro-level biases
– noticing what is different, wishful thinking etc.
• Difficult to know how to deal this in a policy
context

12. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 12
Big Data
• Can give a very detailed picture of a particular
aspect of what is happening
• Showing a broad spectrum of responses and
actors (but biased to those who leave traces)
• Depends upon its availability due to
happenstance of sources, privacy, terms of
service etc.
• Needs extensive pre-processing using complex
algorithms to make sense of it
• (Usually) limited by the biases inherent in the data
• Outcomes usually measured by proxy measures

13. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 13
‘Fat’ data
• ‘Fat’ data is when one has multiple data about the
same set of events, individuals etc.
• Often data can be ‘fattened’ at little extra cost
• For example, if one interviewed some of the
teachers and students involved in a RCT, so that
their responses could be linked to the RCT
results, that would be a fattening of that data
• Key feature: links data from different levels (micro,
meso, macro) and kinds via individual/event IDs
• However, it has been difficult to know how to deal
with such data

14. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 14
A role for simulation models
Part 3:

15. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 15
Agent-Based Simulation
• Is a computer program
• Much like a multi-character game, where each social
actor is represented by a different “agent”
• These agents can each have very different
behaviours and characteristics
• Social phenomena (such as social networks) can
emerge out of the decisions and interaction of these
individual agents (upwards “emergence”)
• But, at the same time, the behaviour of individuals
can be constrained by “downwards” acting rules and
social norms from society and peers
• Can be complicated – avoids strong assumptions

16. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 16
What happens in ABS
• Entities in simulation are decided on
• Behavioural Rules for each agent specified (e.g. sets of
rules like: if this has happened then do this)
• Repeatedly evaluated in parallel to see what happens
• Outcomes are inspected, graphed, pictured, measured
and interpreted in different ways
Simulation
Representations of OutcomesSpecification (incl. rules)

17. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 17
E.G: Schelling’s Segregation Model
Schelling, Thomas C.
1971. Dynamic Models of
Segregation. Journal of
Mathematical Sociology
1:143-186.
Rule: each iteration, each
dot looks at its 8
neighbours and if less than
30% are the same colour
as itself, it moves to a
random empty square
Segregation can result
from wanting only a few
neighbours of a like colour

18. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 18
Using simulation models to integrate
evidence
• Agent-based simulation models are particularly
good at using a variety of evidence, including:
– qualitative micro-level evidence to ensure ‘menu’ of
(behaviours is inclusive enough
– (if available) meso-level surveys or big data might
inform when individuals influence each other and how
– quantitative macro-level evidence to validate/check that
we are not grossly misleading ourselves
• In other words, fat data and simulation analysis
are made for each other
• However, simulation models are not magic – they
do not predict. So what use are they?

19. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 19
Micro-Macro Relationships
Micro/
Individual data Qualitative, behavioural, social psychological data
Theory,
narrative
accounts
Social, economic surveys; Census
Macro/
Social data
Simulation

20. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 20
Using Simulation for Uncertainty
Analysis
• Agent-based simulation models are good at
revealing (and then understanding) otherwise
unexpected outcomes
• That is showing future possibilities – not
probabilities (this is not probabilistic prediction)
• In other words, understanding the ways in which
policies/interventions could go unexpectantly
wrong (or indeed right)
• This allows for monitoring for these emergent
outcomes to be put in place to give the earliest
possible warning to policy actors to adjust policy

21. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 21
An Example: Domestic Water Demand
Part 4:

22. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 22
Social Influence and Domestic Water
Demand (SI&DWD)
• Produced for the Environment Agency/DEFRA
• Part of a bigger project to predict future domestic
water demand in the UK given some different
future politico-economic scenarios and climate
change
• The rest of the project were detailed statistical
models to do the prediction
• This model was to examine the assumptions and
look at the envelope of possibilities
• Joint work with Olivier Barthelemy and Scott Moss

23. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 23
Different kinds of data used
• Actual climate data from the last 40 years
• Detailed statistics of the usage of different kinds
of appliances within the home
• Time series data concerning aggregate domestic
water consumption for clusters of 200-500 houses
• Expert opinion as to the various influences upon
householders as to water usage decisions (but
not the prevalence of these in the population!)

24. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 24
Monthly Water Consumption
REL_CHNG
.88
.75
.63
.50
.38
.25
.13
0.00
-.13
-.25
-.38
-.50
20
10
0
Std.Dev = .17
Mean = .01
N = 81.00

25. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 25
Relative Change in Monthly
Consumption
Date
FEB
2001
SEP
2000
APR
2000
N
O
V
1999
JU
N
1999
JAN
1999
AU
G
1998
M
AR
1998
O
C
T
1997
M
AY
1997
D
EC
1996
JU
L
1996
FEB
1996
SEP
1995
APR
1995
N
O
V
1994
JU
N
1994
REL_CHNG
1.0
.8
.6
.4
.2
-.0
-.2
-.4
-.6

26. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 26
Model Structure - Overall Structure
•Activity
•Frequency
•Volume
Households
Policy
Agent
•Temperature
•Rainfall
•Sunshine
Ground
Aggregate Demand

27. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 27
Model Structure - Microcomponents
• Each household has a variable number of micro-
components (power showers etc.): bath
other_garden_watering shower hand_dishwashing
washing_machine sprinkler clothes_hand_washing
hand_dishwashing toilets sprinkler power_shower
• Actions are expressed by the frequency and
volume of use of each microcomponent
• AVF distribution in model calibrated by data from
the Three Valleys

28. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 28
An Example Social Structure
- Global Biased
- Locally Biased
- Self Biased

29. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 29
Policy Agent - Behaviour
• After the first month of dry conditions, suggests
AFV actions to all households
• These actions are then included in the list of those
considered by the households
• If the household’s weights predispose it, it may
decide to adopt these actions
• Some other neighbours might imitate these
actions etc.

30. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 30
Number of consecutive dry months in
historical scenario
0
1
2
3
4
5
6
7
8
9
J-73
J-74
J-75
J-76
J-77
J-78
J-79
J-80
J-81
J-82
J-83
J-84
J-85
J-86
J-87
J-88
J-89
J-90
J-91
J-92
J-93
J-94
J-95
J-96
J-97
Simulation Date
Numberofconsequativedrymonths

31. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 31
Simulated Monthly Water
Consumption
REL_CHNG
.075
.063
.050
.037
.025
.012
-.000
-.013
-.025
-.038
-.050
120
100
80
60
40
20
0
Std. Dev = .01
Mean= -.000
N = 325.00

32. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 32
Monthly Water Consumption (again)
REL_CHNG
.88
.75
.63
.50
.38
.25
.13
0.00
-.13
-.25
-.38
-.50
20
10
0
Std.Dev = .17
Mean = .01
N = 81.00

33. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 33
Simulated Change in Monthly
Consumption
Date
SEP
1997
APR
1996
N
O
V
1994
JU
N
1993
JAN
1992
AU
G
1990
M
AR
1989
O
C
T
1987
M
AY
1986
D
EC
1984
JU
L
1983
FE
B
1982
SEP
1980
APR
1979
N
O
V
1977
JU
N
1976
JAN
1975
AU
G
1973
M
AR
1972
O
C
T
1970
REL_CHNG
.10
.08
.06
.04
.02
0.00
-.02
-.04
-.06

34. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 34
Relative Change in Monthly
Consumption (again)
Date
FEB
2001
SEP
2000
APR
2000
N
O
V
1999
JU
N
1999
JAN
1999
AU
G
1998
M
AR
1998
O
C
T
1997
M
AY
1997
D
EC
1996
JU
L
1996
FEB
1996
SEP
1995
APR
1995
N
O
V
1994
JU
N
1994
REL_CHNG
1.0
.8
.6
.4
.2
-.0
-.2
-.4
-.6

35. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 35
30% Neigh. biased, historical
scenario, historical innov. datesAggregate demand series scaled so 1973=100
0
20
40
60
80
100
120
140
160
180
200
J-
73
J-
74
J-
75
J-
76
J-
77
J-
78
J-
79
J-
80
J-
81
J-
82
J-
83
J-
84
J-
85
J-
86
J-
87
J-
88
J-
89
J-
90
J-
91
J-
92
J-
93
J-
94
J-
95
J-
96
J-
97
Simulation Date
RelativeDemand

36. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 36
80% Neigh. biased, historical
scenario, historical innov. datesAggregate demand series scaled so 1973=100
0
20
40
60
80
100
120
140
160
180
200
J-
73
J-
74
J-
75
J-
76
J-
77
J-
78
J-
79
J-
80
J-
81
J-
82
J-
83
J-
84
J-
85
J-
86
J-
87
J-
88
J-
89
J-
90
J-
91
J-
92
J-
93
J-
94
J-
95
J-
96
J-
97
Simulation Date
RelativeDemand

37. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 37
80% Neigh. biased, medium-high
scenario, historical innov. DatesAggregate demand series scaled so 1973=100
0
20
40
60
80
100
120
140
160
180
200
Jan-
73
Jan-
74
Jan-
75
Jan-
76
Jan-
77
Jan-
78
Jan-
79
Jan-
80
Jan-
81
Jan-
82
Jan-
83
Jan-
84
Jan-
85
Jan-
86
Jan-
87
Jan-
88
Jan-
89
Jan-
90
Jan-
91
Jan-
92
Jan-
93
Jan-
94
Jan-
95
Jan-
96
Jan-
97
Simulation Date
RelativeDemand

38. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 38
What did the model tell us?
• That it is possible that social processes:
– can cause a high and unpredictable variety in patterns
of demand
– can ‘lock-in’ behavioural patterns and partially ‘insulate’
them from outside influence (droughts only occasionally
had a permenant affect on patterns of consumption)
• The availability of new products could dominate
effects from changing consumptions habits
• Downing, T.E, et al. (2003). Climate Change and the
Demand for Water, Research Report, Stockholm
Environment Institute Oxford Office, Oxford.
(http://www.sei.se/oxford/ccdew/)

39. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 39
Conclusions and Implications
Part 5:

40. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 40
Summary
• In many kinds of situation we can not predict –
even approximately. This makes traditional policy
evaluation impossible
• All kinds of data have their disadvantages singly
• ‘Fattened’ data can be used using simulation
analysis to understand the possible ways in which
a policy/intervention might work out
• Monitors (e.g. ‘live’ data visualisations) can be
designed based on this understanding to allow
policy to be better ‘driven’ and for better-informed
contingency planning to done

41. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 41
Towards Adaptive Policy
• Where impacts of policy/interventions/nudges can
be predicted continue as before…
• But in other (complex) situations:
1. Collect enriched (‘fat’) data (including from
stakeholders)
2. Understand some of the ways a policy could
go wrong (maybe using simulation analyses)
and implement monitors for these possibilities
3. Adapt policy rapidly based on monitors and
knowledge of some of the emergent
possibilities

42. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 42
The End
Manchester Metropolitan University
http://mmu.ac.uk
Centre for Policy Modelling
http://cfpm.org
Slides are Available at:
http://slideshare.net/BruceEdmonds
@

43. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 43
Understanding Voter Turnout
Additional Example 1:

44. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 44
The Social Complexity of Immigration and Diversity was a 5-year project
with the Institute for Social Change and the Department of Theoretical
Physics at University of Manchester. It was funded under the “Complexity
Science for the Real World” initiative of the EPSRC from July 2010 to Jan
2016.
The idea of SCID was to apply the techniques and tools of complexity
science to real world issues, in this case of immigration and diversity. The
example here focuses on the issue why people bother to vote and how the
mix of ethnicities and immigration might impact upon this.
Copy of Project Website:
http://cfpm.org/scid

45. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 45
Model ‘layers’ and the different kinds of
data used to support these
Underlying data about
population composition
Demographics of people in
households
Social network formation and
maintenance (homophily)
Influence via social networks
• Political discussions
Voting Behaviour

46. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 46
Discuss-politics-with person-23 blue expert=false
neighbour-network year=10 month=3
Lots-family-discussions year=10 month=2
Etc.
Memory
Level-of-Political-Interest
Age
Ethnicity
Class
Activities
AHousehold
An Agent’s Memory of Events
Etc.
Changing personal
networks over which
social influence occurs
Composed of households of
individuals initialised from
detailed survey data
Each agent has a rich variety of
individual (heterogeneous)
characteristics
Including a (fallible) memory of
events and influences

47. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 47
Evidence schema
Qualitative
Intuitive
understanding
Observations
of Phenomena
Quantitative
Models
Data
Text from interviewsTime Series Data etc.
A-B Simulation ‘Causal Stories’
Expert OpinionConclusions

48. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 48
Example Output: why do people vote (if
they do)
Intervention: voter
mobilisation
Effect: on civic
duty norms Effect: on habit-
based behaviour
Time
%ofvotersbyreason

49. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 49
Simulated Social Network at 1950
Established
immigrants: Irish,
WWII Polish etc.
Majority: longstanding
ethnicities
Newer
immigrants

50. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 50
Simulated Social Network at 2010

51. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 51
What did we learn from this?
• Previous work claimed that persuading people to
vote would have a ‘cascade’ effect, resulting in
much more participation – the simulation showed
that such a claim might require very strong
assumptions
• The structure of the social network and
immigration can matter (e.g. if immigrants come in
as individuals or families)
• That the network is dynamic might also matter
and affect outcomes

52. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 52
Fishing
Additional Example 2:

53. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 53
Socio-Ecological Systems
• Are the combination of human society embedded
within an ecological system (SES)
• Many social and ecological systems are far too
complex to predict
• Their combination is doubly complex
• E.g. fisheries, deforestation, species extinctions
• Yet we still basically use the 1970s robotics
“predict and plan” approach to these…
• …as if we can plan optimum policies by
estimating/projecting future impact

54. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 54
Why simple ecological models won’t
predict levels of fish
• Simpler models do not necessarily get things
“roughly” right
• Simpler models are not more general
• They can also be very deceptive – especially with
regards to complex ways things can go wrong
• In complex systems the detailed interactions can
take outcomes ‘far from equilibrium’ and far from
average behaviour
• Sometimes, with complex systems, a simple
model that relies on strong assumptions can be
far worse than having no models at all

55. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 55
A Cautionary Tale
• On the 2nd July 1992 Canada’s fisheries minister,
placed a moratorium on all cod fishing off
Newfoundland. That day 30,000 people lost their jobs.
• Scientists and fisheries departments throughout much
of the 1980s estimated a 15% annual rate of growth in
the stock – (figures that were consistently disputed by
inshore fishermen).
• The subsequent Harris Report (1992) said (among
many other things) that: “..scientists, lulled by false
data signals and… overconfident of the validity of
their predictions, failed to recognize the statistical
inadequacies in … [their] model[s] and failed to …
recognize the high risk involved with state-of-stock
advice based on … unreliable data series.”

56. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 56
The Model
Allows for a complex,
multi-species grid of
ecologies to evolve
This is highly dynamic,
showing a continual
changing balance
between species
There is: movement between patches, complex and
dynamic predation between (non-plant) species,
competition for resources, slow evolution
"A test-bed ecological model”. CoMSES Computational Model Library.
Documentation and code available via: http://comses.net

57. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 57
This version designed to test possible
outcomes of fishing policies
• Complex aquatic plant ecology of ~30 species
evolved from scratch
• Herbivore fish injected into ecology and whole
system further evolved (to 40-70 species)
• Once a complex ecology with higher-order
predators then system is fixed as starting point
• Then possible impacts of fishing policies can be
examined and understood

58. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 58
Total Extinction Prob. & Av. Total Harvest
(last 100 ticks) for different catch levels
Catch level (per tick)
ProportionofMaximum

59. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 59
Num Fish (all species, 20 runs) – catch
level 25 – each line is a different model run
0
1000
2000
3000
4000
5000
6000
0
31
62
93
124
155
186
217
248
279
310
341
372
403
434
465
496
527
558
589
620
651
682
713
744
775
806
837
868
899
930
961
992

60. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 60
Num Fish (all species, 20 runs) – catch
level 35 – each line is a different model run
0
1000
2000
3000
4000
5000
6000
0
31
62
93
124
155
186
217
248
279
310
341
372
403
434
465
496
527
558
589
620
651
682
713
744
775
806
837
868
899
930
961
992
Catch target=30

61. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 61
Average Number of Species vs. Catch
Level (from a different starting ecology)
0
2
4
6
8
10
12
14
0 5 10 15 20 25 30 35 40
Num Species Fish

62. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 62
Average Number of Species, Catch=30
0
5
10
15
20
25
30
35
0 200 400 600 800 1000
AverageNumberofSpecies
Time
"by patches"
"uniform"

63. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 63
Average Number of Species, Catch=40
0
5
10
15
20
25
30
35
0 200 400 600 800 1000
AverageNumberofSpecies
Time
"by patches"
"uniform"

64. Mixing fat data, simulation and policy - what could possibly go wrong?, Bruce Edmonds, CECAN, London, Feb. 2019. slide 64
A risk-analysis approach
1. Give up on estimating future impact or “safe”
levels of exploitation
2. Make simulation models that include more of the
observed complication and complex interactions
3. Run these lots of times with various scenarios to
discover some of the ways in which things can
go surprisingly wrong (or surprisingly right)
4. Put in place sensors/measures that would give
us the earliest possible warning that these might
be occurring in real life
5. React quickly if these warning emerge