Louise Shaxson Associate, Delta Partnership Impact & insight workshop King’s College London 25 October 2007 Practical tools for evidence based policy making: DEVELOPING LINES OF ARGUMENT

Acknowledgements This presentation draws on my work for Defra, 2004-ongoing: it is my interpretation rather than any statement of Defra approach or policy. However, key reference documents include: Our Approach to Evidence & Innovation (from www.defra.gov.uk ) Defra’s pages on evidence-based policymaking (from www.defra.gov.uk ) Bielak et al (forthcoming, 2008) – “From science communications to knowledge brokering: the shift from ‘science push’ to ‘policy pull ’”. Chapter ? in Science Communication in Social Context: Strategies for the Future . Springer.

This presentation draws on my work for Defra, 2004-ongoing: it is my interpretation rather than any statement of Defra approach or policy. However, key reference documents include:

Our Approach to Evidence & Innovation (from www.defra.gov.uk )

Defra’s pages on evidence-based policymaking (from www.defra.gov.uk )

Bielak et al (forthcoming, 2008) – “From science communications to knowledge brokering: the shift from ‘science push’ to ‘policy pull ’”. Chapter ? in Science Communication in Social Context: Strategies for the Future . Springer.

Policy trajectory – external drivers Complex policy goals are nebulous, contestable, open to multiple interpretations: viz ‘sustainability’ Future evidence needs are interdisciplinary, overlapping, and constantly emerging Government’s role in fostering innovation is not well understood Competing pressures to open up the policy process and improve analytical rigour – tendency is to projectise the former and programme the latter.

Complex policy goals are nebulous, contestable, open to multiple interpretations: viz ‘sustainability’

Future evidence needs are interdisciplinary, overlapping, and constantly emerging

Government’s role in fostering innovation is not well understood

Competing pressures to open up the policy process and improve analytical rigour – tendency is to projectise the former and programme the latter.

Three levels of policy work (Defra, 2007) STRATEGY … setting the direction of travel by understanding and scoping existing and future context and challenges and developing capability and capacity to deliver desired outcomes POLICY DEVELOPMENT … structuring choice for decision-makers that is based on robust evidence POLICY DELIVERY … implementing policies with stakeholders to deliver measurable outputs

STRATEGY … setting the direction of travel by understanding and scoping existing and future context and challenges and developing capability and capacity to deliver desired outcomes

POLICY DEVELOPMENT … structuring choice for decision-makers that is based on robust evidence

POLICY DELIVERY … implementing policies with stakeholders to deliver measurable outputs

Evidence for policy is… … any robust information that helps to turn a Department’s strategic priorities into something concrete, manageable and achievable. But… … there are multiple criteria for robustness, which means that there is no single gold standard: it is negotiable and open to challenge & alternative interpretation. This in turn means that… … robustness in the evidence base includes best practice in stakeholder engagement. Evidence from stakeholders isn’t something you use to test what you think you know.

… any robust information that helps to turn a Department’s strategic priorities into something concrete, manageable and achievable. But…

… there are multiple criteria for robustness, which means that there is no single gold standard: it is negotiable and open to challenge & alternative interpretation. This in turn means that…

… robustness in the evidence base includes best practice in stakeholder engagement. Evidence from stakeholders isn’t something you use to test what you think you know.

Sustainable Consumption & Production One of four priority areas for action set out by Defra in 2005 New products & services with lower environmental impacts across their lifecycle; new business models; new approaches to encouraging consumer behaviour change Little certainty of scale of challenge: in time or globally SCP is an influencing, rather than a delivery programme: others hold the policy levers Our question – how do we construct an evidence base for SCP policy?

One of four priority areas for action set out by Defra in 2005

New products & services with lower environmental impacts across their lifecycle; new business models; new approaches to encouraging consumer behaviour change

Little certainty of scale of challenge: in time or globally

SCP is an influencing, rather than a delivery programme: others hold the policy levers

Our question – how do we construct an evidence base for SCP policy?

Policy trajectory: internal drivers Smaller policy core (Gershon & related processes) – constant downward pressure on central Govt budgets; Policymakers have very limited opportunities to present their work to Ministers – complicated by Cabinet reshuffles; How can we provide sufficient breadth of evidence to Ministerial discussions of the policy landscape & encourage rigorous analysis of its alternative interpretations when the reality is that time pressures drive us towards narrow channels of problem-specific questions?

Smaller policy core (Gershon & related processes) – constant downward pressure on central Govt budgets;

Policymakers have very limited opportunities to present their work to Ministers – complicated by Cabinet reshuffles;

How can we provide sufficient breadth of evidence to Ministerial discussions of the policy landscape & encourage rigorous analysis of its alternative interpretations when the reality is that time pressures drive us towards narrow channels of problem-specific questions?

Lines of argument Evidence for policy emerges from : Data Analytical evidence Views from stakeholder engagement Engage with stakeholders in a structured way to establish a ‘line of argument’ between: their particular goal definition the values inherent in how they define the goal the evidence they believe will validate their conviction that this is the path policy should take

Evidence for policy emerges from :

Data

Analytical evidence

Views from stakeholder engagement

Engage with stakeholders in a structured way to establish a ‘line of argument’ between:

their particular goal definition

the values inherent in how they define the goal

the evidence they believe will validate their conviction that this is the path policy should take

How interdisciplinary can you get? In developing a method to draw out the lines of argument we drew from: Science policy (Stirling, Jasanoff, Gibbons et al …) Stakeholder engagement (Chambers onwards…) Political science (Rayner…) Knowledge management (Snowden: Cynefin) Organisational behaviour (IBM expertise)

In developing a method to draw out the lines of argument we drew from:

Science policy (Stirling, Jasanoff, Gibbons et al …)

Stakeholder engagement (Chambers onwards…)

Political science (Rayner…)

Knowledge management (Snowden: Cynefin)

Organisational behaviour (IBM expertise)

‘ Five Whys’ Why is this issue important? Why is change happening? Why do we need to intervene in the change process (to change the rate or scale of change)? Why should Government intervene in the change process? Summarise into: Why do we need a policy on this issue? Use this to underpin the question: What evidence do we need to collect to inform policy development on this issue?

Why is this issue important?

Why is change happening?

Why do we need to intervene in the change process (to change the rate or scale of change)?

Why should Government intervene in the change process?

Summarise into:

Why do we need a policy on this issue?

Use this to underpin the question:

What evidence do we need to collect to inform policy development on this issue?

How well did we do (1)? We evaluated the technique internally, concluding that it: Is a cost-effective yet powerful method of scoping the evidence base for policy way Allowed multiple and competing goals to co-exist, valuing dissent & alternative interpretation Broadened thinking about innovation Communicated complex messages into policy Communicated the policy questions to external stakeholders

We evaluated the technique internally, concluding that it:

Is a cost-effective yet powerful method of scoping the evidence base for policy way

Allowed multiple and competing goals to co-exist, valuing dissent & alternative interpretation

Broadened thinking about innovation

Communicated complex messages into policy

Communicated the policy questions to external stakeholders

How well did we do (2)? Real value in scoping the evidence base for policy: for new policy areas (little evidence) existing policy areas (surprisingly little policy-relevant evidence!) changed policy goals (how well-aligned is available evidence?) more strategic thinking about evidence needs for policy By engendering rigorous dialogue, lines of argument: Show policymakers that challenge and alternative interpretation is a inherent part of the process – doesn’t encourage a false consensus on the issues Structure choice for decision-makers (see previously) Helps put the politics back into policymaking?

Real value in scoping the evidence base for policy: for

new policy areas (little evidence)

existing policy areas (surprisingly little policy-relevant evidence!)

changed policy goals (how well-aligned is available evidence?)

more strategic thinking about evidence needs for policy

By engendering rigorous dialogue, lines of argument:

Show policymakers that challenge and alternative interpretation is a inherent part of the process – doesn’t encourage a false consensus on the issues

Structure choice for decision-makers (see previously)

Helps put the politics back into policymaking?

Do it yourself… We would have conducted a backcasting exercise… which might or might not have given us the goal of… Reducing the global environmental footprint of the UK’s production and consumption patterns

We would have conducted a backcasting exercise… which might or might not have given us the goal of…

Reducing the global environmental footprint of the UK’s production and consumption patterns