Bruce Edmonds gave a talk on policy making using modeling in complex systems. He argued that traditional approaches to modeling and policy making do not work for complex systems, which require different modeling techniques and relationships between modelers and policy actors. Specifically, he noted that complex systems cannot be accurately forecast, may not reach equilibriums, and require iterative modeling to understand possible outcomes rather than predict impacts. He suggested moving from probabilistic forecasting to risk analysis of different possibilities to better inform adaptive policy making.

1. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 1
Policy Making using Modelling in a
Complex world
Bruce Edmonds
Centre for Policy Modelling
Manchester Metropolitan University

2. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 2
or… towards what might be in a ‘Cyan Book’
(halfway between the aqua and green books)

3. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 3
Introduction
Part 0

4. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 4
Simple systems…
… may be complicated but behave in predictable
ways, allowing them to be represented by models...
• where one can use them to numerically forecast
• where uncertainty can be analytically estimated
• where one can get rough estimates cheaply, and
better estimates with increasing investment
• which one can sensibly plan and execute ot
systematically
• where there is a basically one right way of doing it
• so that one can fully understand the model

5. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 5
A double pendulum
Even with only two bits of wood the result can be complex
An Example Video is at:
http://www.youtube.com/watch?v=czLIj-4suOk
A trace of the motion:

6. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 6
The Main Point of the Talk…
…is that complex systems need to be dealt with in a
different way to that of simple systems...
...not only modelled in a different way using different
techniques but also how models about complex
systems are used in any policy development
process needs to change.
• Complex modelling will be increasingly important
as we try to develop better policies and deal with
complex and fast moving situations
• But it can not be ‘business as usual’ – just doing
better modelling with the same modeller–policy
actor relationship will not work well

7. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 7
Structure of the (rest of the) Talk
1. A bit about modelling context, purposes
and tensions
2. Some of the underlying assumptions
and habits that need to change
3. An example model – Stefano Picascia’s
Modelling of the Housing Rental Market
4. Some suggestions as to ways forward

8. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 8
Modelling, its context, purposes and
tensions
Part 1

9. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 9
Some of the Context
• An increasingly professionalised coterie of skilled
modellers within government (GORS etc.)
• Leading to new standards for modelling and
analysis within government – the ‘Aqua Book’ –
which lays down very sensible guidelines for
model development and quality assurance
• Within the context of the ‘Green Book’ – a
framework for policy development and evaluation
• Along side a set of academic and other experts
increasingly willing to be involved in helping
government model and analyse complex stuff

10. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 10
Different modelling purposes
Models can be used for a wide variety of different
purposes, and these impact upon the kind of
techniques needed and its difficulties, e.g.
• Forecasting – predicting unknown (e.g. future)
situations and outcomes
• Explanation – understanding how known
outcomes might have come about
• Theoretical Exploration – understanding a
complex model by exploring some of its properties
and behaviours
• Analogy – using a model as a way of thinking
about something else

11. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 11
A picture of modelling
whatisobservedor
measured
themodel
themodellers
themodelusers

12. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 12
Model Scope
• The scope of a model is the conditions under which it
is useful for its planned purpose
• Whilst this is implicit and stable for many simple
systems, this is not the case for many complex ones
• Thus trying to make scope explicit is important, and
these relate to model assumptions
• A process not modelled (and hence outside its scope)
can overwhelm the results…
• ..but in complex systems internal processes of
change can also emerge, and some of these can be
usefully modelled (but only in more complex ways)

13. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 13
Possible modelling trade-offs
• Some desiderata for
models: validity,
formality, simplicity and
generality
• these are difficult to
obtain simultaneously
(for complex systems)
• there is some sort of
complicated trade-off
between them (for each
modelling exercise)
simplicity
generality
validity
formality
Analogy
Solvable
Mathematical
Model
Data
What
Policy
Actors
Want

14. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 14
Some underlying assumptions and
habits
Part 2

15. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 15
The Green Book
• Basically a formulation of cost-benefit analysis
• As if an economist had written a manual for policy
actors in how to think (i.e. as their theory states)
• It does have some useful advice on generating
and considering alternatives and trying to judge
uncertainty
But does assume that one can:
1. list the main alternative options
2. forecast the results of these
3. put meaningful numerical values on these

16. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 16
Quantification
• Implicit throughout the Green Book approach
• Makes life much easier for policy actors
• Especially when asked to justify an approach
• But can be more misleading than helpful because it
gives a false impression of accuracy
• And implicitly leads to a focus on the measurable and
that things will ‘average out’ etc.
• Was a limitation of purely mathematical approaches,
but computer simulation does not have to be focused
on these aspects
• Collecting qualitatively different possible outcomes
might be much more useful

17. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 17
No gradual approximation, but
scope-limited usefulness
It is often assumed that as time and effort increase
the accuracy of the results improve, but this is not
the case with complex systems and models
Rather in order for the outcomes to be within scope
enough iterative development has to occur
Before this the results are worse than nothing
Time and cost
Error

18. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 18
That (real, complex) systems ever get
near an equilibrium
• From economics, there is an obsession with
equilibrium in much modelling
• Where, even when it is known equilibria are not
observed, they are assumed in order to forecast
• Most macro economic models (and many cost-based
planning models) do this
• In complex systems, even when they are in near-
equilibrium state, they can move away from this
• Good when things are not changing very much,
almost useless for turning points or when structural
change is occurring
• (See forthcoming open-access book “Non-Equilibrium
Social Science”)

19. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 19
The Aqua Book
• Is a standard for the development and quality
assurance of modelling/analysis for policy
• Has a lot of very sensible advice (I wish that my
academic colleagues had similar standards)
• It distinguishes different roles involved in the
modelling process, which clarify the chain of
command/assurance and recommends extensive and
frequent communication between them
• However, it only talks about the responsibilities of the
modelling team, as if modelling can be an ‘on demand
service’ for policy actors
• Mostly it is written for the cases where the model or
situation is basically simple
• See my recent review in JASSS (jasss.surrey.ac.uk)

20. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 20
That different models should agree
• Sometimes policy actors complain that different
models of the same phenomena disagree
• But this is inevitable where different models are
taking different approaches and making different
assumptions – the results are relative to these
• One can try and force models to agree, but in the
process one eliminates the variety of modelling
assumptions (in return for an illusion of certainty)
• In risk-analyses one wants variety so that it is
more likely that future possibilities can be
anticipated

21. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 21
Planning and Managing Modelling
• In a simple case one can apply an approach (as
described in the Aqua book) where one carefully
plans, manages and evaluates models
• As if this was like building a bridge!
• But in complex cases complications about what
needs to be included or not requires a more
iterative approach…
• ...where models are repeatedly built for a purpose
and the lessons learnt as you go along...
• Becuase the difficulties can not be predicted in
complex cases!

22. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 22
Compartmentalism
• That some problems can be separated into
smaller sub-problems which can be modelled
more simply
• Not true in many complex cases, where the scope
of modelling is dependent on having enough of
the key processes represented
• Sometimes several different modelling
approaches with different (but overlapping)
assumptions can be more helpful
• Just fiddling, incrementally expanding an existing
(and failing) model will probably not help here

23. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 23
Instrumental Institutionalisation
• Even a wrong or clearly inadequate model can be a
useful part of a policy development process if the
model is flexible enough
• Because when used repeatedly its limitations and
oddities can be fixed with institutional ‘kludges’
• Becoming something more like a consistent ‘base
line’ from which policy can be debated
• e.g. the standard transport models
• But as these are instutionalised, they are increasingly
difficult to change because the other policy processes
have adapted to it
• Resulting in a situation similar to the apocryphal
‘slowly boiled frog’ story – at each time the motivation
to change is less than the perceived advantage of
doing so, in which case wait for a crisis to occur!

24. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 24
An Example: A Model of the Rental
Housing Market
Part 3

25. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 25
The model
• By Stefano Picascia, an PhD student of mine,
now at Sienna University, Italy
• Is an agent-based simulation that represents both
tenants and developers co-adapting
• Is geographically based with tenants making
decisions as where to move to based on location
as well as quality of housing and price
• Developers put in captial to build/rennovate
housing for tenants
• Rents are determined by the quality and prices of
surrounding housing

26. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 26
The Manchester Case
Waves of price
changes can
spread
Can have different
outcomes each
time it is run
Has also been
applied to London
and Beirut
Video of model running is at:
http://www.youtube.com/watch?v=PtYTtkPrACM

27. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 27
Average prices in a run

28. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 28
Different Sectors of the City in a run

29. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 29
What it does and does not tell us
In the model (which is the private rental sector only):
• That change is fundamentally internally driven as
well as due to outside events
• Price oscillations are endemic to the system
• That some regions of cities will be stuck as low
quality housing for long periods of time
• The very high price regions stay that way
• That under certain conditions sudden
‘gentrification’ may occur to some degree
• For poorer districts decline is gradual and
continual between any such periods

30. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 30
Some ways forward and conclusions
for complex cases
Part 4

31. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 31
From Probabilistic to Possibilistic
• When outcomes can not be sensibly forecast…
• And especially numerically forecast…
• …where even probability zones or 90% bounds
are misleading
• Then moving to an approach that models and
understand (more of) underlying processes...
• ...in terms of the different kinds of outcome might
be much more informative
• Each outcome tagged with its own assumptions
and scopes (if they differ)

32. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 32
From Forecasting to Risk Analysis
• However much one might like forecasting, often it is simply
not possible…
• ...let alone in a way such that the outcomes from different
options can be compared!
• Predicting outcomes can be more misleading than helpful
• Rather it may be more approapriate to use models for risk
analysis – finding all the ways a policy might go wrong (or
right!)
• Techniques are available to help discover and understand
how endogenous processes might result in different future
possibilities
• Which can then inform the design of ‘early warning’
monitors giving the most immediate feedback to policy
makers

33. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 33
Informing the adaptive ‘driving’ of
policy
• Complex models are no good for policy makers!
• Because they have to make decisions on grounds
they understand and know the reliability of
• They can not (and should not) delegate this to
‘experts’ and their inscrutable models
• Rather modellers should use their modelling to
understand the key emergent kinds of outcome
• To inform:
– the consideration of these kinds of outcome
– the design of appropriate data visalisations
– the design of ‘earl warning indicators
• …So that policy can adapt to changing trends and
events as quickly and fluidly as possible

34. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 34
Conclusions
• Modelling of complex phenomena is not cheap or
quick and requires iterative development
• It will not forecast the impact of potential policies
or events, but can anticipate possible future
outcomes in a way intuition can not
• There will always be a ‘scope’ – a set of
conditions/assumptions a model depends upon
• But a good model can repay its investment in
terms of cost and improving people’s lives many,
many times over

35. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 35
Summary
It is no good wishing that the world or
modelling is simple and trying to ‘force’ it to
be so, one has to adapt to suit reality…
…this includes how models and modelling
are used by the policy process

36. Policy Making using Modelling in a Complex world, Bruce Edmonds, CECAN, London, 20th July 2016. slide 36
The End
The Centre for Policy Modelling:
http://cfpm.org
These slides will be available at: http://slideshare.net/BruceEdmonds
Stefano’s work was
developed under this
project, funded by
the EPSRC, grant
number EP/H02171X